I really suggest you watch 3Blue1Brown’s series on the subject. Watch through all of them. Don’t skip anything.

One of the best series I’ve seen anywhere. Check them out!

But what *is* a Neural Network? | Deep learning, chapter 1
Gradient descent, how neural networks learn | Deep learning, chapter 2
What is backpropagation and what is it actually doing? | Deep learning, chapter 3
Backpropagation calculus | Appendix to deep learning chapter 3

The post Wicked cool neural network video primer appeared first on Pietari Heino's personal website.

1. I opened the pdf
2. I clicked sign
3. They sent me an SMS with a link that opened a “sign here” web page
4. I signed, refreshed the original web page, and vóila, DocHub let me attach the signature to the document, resize it, change the colouring and so forth

Like seriously — they just sent me a text and I opened the link. I didn’t sign in to any page on my phone. I didn’t sign in to Google, to DocHub, to anything. It just worked.

One of those moments when you really feel that the engineering work is impeccable and so flawlessly superior to the everyday sw one usually encounters. So good.

The post DocHub: so smooth appeared first on Pietari Heino's personal website.

I really hope you read the paper, it’s wicked fascinating software engineering. I liked the piece so much that I decided to read it again and write some notes/summaries of the different topics touched. Scroll through them below and read the paper (which is vastly more infromative than what you can find here!).

Super-shortly:
Pros: unified versioning, extensive code sharing, simplified dependency management, atomic changes, large-scale refactoring, collaboration across teams, flexible code ownership, code visibility
Cons: having to create *and* scale tools for development and exectuion and maintain code health (also a possibility of potential codebase complexity)

The repo:
~1 billion files
~35 million commits
~85 TB of data
~2 billion lines of code
~9 million source files

2014: 15 million lines of code changed in 250,000 files. 25,000 users and avg 500,000 queries per second.
note: most of the traffic comes from Google’s automated build and test systems

Compare to Linux kernel: ~15 million lines of code in ~40 000 files

Google Piper design
– stores a single large repository
– implemented on top of standard Google infra, namely Spanner
– distributed on 10 datacenters
– Paxos for replica consistency
– Google infra and private networks cut the latency and deliver needed speed
– Google originally used a massive Perforce instance with custom-built caching and other infra for over 10 years

Piper security
– supports file-level access control lists
– most of the stuff seen by everyone, anything may be hidden if needed
– read/write logs; owner can see who viewed, when, and what
– purgin of accidental critical secrets
– for instance business critical secrets like algorithms might not be available for everyone (but: over 99 % of all version-controlled stuff is seen by all full-time Googlers)

Piper workflow
– create a local copy, store files in the developer’s workspace
– – this is like working copy in Subversion, local clone in Git, or client in Perforce
– pull updates from Piper
– share the workspace as a snapshot for other devs to review
– commit *only* after code-review

Clients in the Cloud, or CitC
– cloud-based storage backend + Linux-only FUSE fs
– Piper workspaces seen as directories in the fs
– support the usual Unix tools
– local changes laid on top
– browsing, searching, editing any files in the Piper repo
– only edited files stored locally
– avg workspace has <10 files while still showing everything in the Piper repo
– *all writes* stored automatically, can be tagged, named, and rollbacked

Trunk-based development
– vast majority of Piper users work on “head”, “trunk”, or “mainline”, that is the most recent version of everything
– all commits in there
– all changes seen by everyone using Piper after every commit (remember: commits only after code-review)
– using branches very very rare except for releases
– releases usually a snapshot of the trunk + cherry-picks from it
– no dev branches, no feature branches, no nothing
– feature-development through the use of feature-flags in code
– feature-flags controlled by conf. files, no need for new binaries
– feature-flags typically used in project-specific code, not libraries
– easy to experiment with small amount of users

Code review
– nothing is committed without a code review
– the committer can enable a flag for auto-commit if the review passes
– the reviewers have tools for viewing and adjusting the code easily anywhere in the Piper repo (tools are named Critique and CodeSearch)
– commits have to be accepted by directory owners
– remember: the whole Piper repo is availabe for anyone -> anyone can propose changes in any piece of code anywhere, but the owners of directories have to accept them
– directory owners are the people most familiar with the code/project/library in question

Commit-infra & refactoring
– automatic rebuild of all dependencies, testing
– automatic rollback in case of widespread breakage
– vast and customizable pre-submit testing and analysis, runs before anything is committed
– static analysis system called Tricorder
– – provides data on code quality, test coverage, test results
– – provides automatic suggestions for fixes with one-click applying
– – triggered after all changes and periodically
– – used to ensure codebase health
– set of devs periodically dig through Piper directories to refactor code in order to keep it healthy
– large backwards-compatible changes first, removing unused paths second
– tool called Rosie suppors that by splitting the large patches made by the devs into smaller patches that are individually reviewed by the directory owners

Analysis
Advantages
– unified versioning, one source of truth
– extensive code-sharing and reuse
– simplified dependency management
– atomic changes
– large-scale refactoring
– collaboration across teams
– flexible team boundaries, code ownership and visibility, implicit namespacing
– all code depend on other code directly
– the diamond-dependency problem is gone
– atomic changes enable refactorings of variables or api calls for hundreds of thousands of files without test/build breakage (in a single commit)
– engineers don’t depend on specific versions -> no need to update them
– all files uniquely identified
– a good example:
– – the Google compiler team can run regression etc. tests nightly on all affected code and validate new versions
– – code can be refactored to support new versions of compilers before shipping them
– – ~20 compiler releases a year
– – compilers can be tuned to use best possible default settings

Drawbacks, trade-offs etc.
– tooling investment is HUGE
– couldn’t be used without all the special support-systems
– codebase complexity,
– unnecessary dependencies
– discoverity difficulties
– effort in code health
– sometimes hard to explore code
– the usual suspects like grep unusable from time to time
– too easy to add dependencies -> unused dependencies
– lack of will to write documentation if everyone can look up the apis themselves
– depending on more than just the api because you see how the code works

Alternatives
– the favoring and use of DVCSs has have grown -> moving has been investigated
– moving to a DVCS (eg Git) would require a split to thousands of repos
– – Android is Git hosted and North of 800 repos
– currently available DCVSs don’t provide needed security controls
– investigating whether Mercurial could be made to support Google scale

Checkout the wonderful paper here.

The post Google’s ultra-large-scale monolithic source code repository appeared first on Pietari Heino's personal website.

The first gives you a deep-dive into the inner-workings of widely used Deflate algorithm and the second is a piece from Facebook’s dev blog introducing their new data compression algorithm Zstandard.

Start with the first one.

The post Interesting reading: data compression from Deflate to Zstandard appeared first on Pietari Heino's personal website.

So back in the day CPU architectures introduced the stack in order to allow better control of the programs’ execution flow and for instance make recursion possible (if it’s not clear for you, read this post and try to come up with the reason why you cannot have recursion without the stack!). Nowadays stack is a must and you cannot find a piece of hardware executing software without a stack (as far as I know). The stack is interesting. It grows with the push operations and decreases with the pop operations. Every time the CPU calls a new function the stack is grown and every time the execution of a function is over the stack is decreased. What goes in there, specifically, are the local variables for a function and the return address to use after the execution. So lets say the CPU calls a void function called count that is supposed to print numbers from 0 to 10. What we push to the stack is the return address which is the address of the next instruction that is supposed to be executed right after the function is done and space for the count’s local variables (it supposedly needs a loop and a loop variable). When count is done, we decrease the stack with pop operation which then gets rid off the count’s local variables and returns the execution to the location pushed to the stack earlier. It grows dynamically, so to say, and the whole thing is really interesting, the way it works and the cleverness of it. It’s automatically grown and decreased and always keeps each functions’ environments intact and stores the addresses for continuing execution. It’s very simple but really powerful at the same time.

Stack exploiting trampoline, (c) Ars Technica

The problem is… what if an adversary party could write some malicious code into your stack and override the bits that holds the information where to go next? Specifically, overwrite the return address from the stack with a return address of the party’s own. That would inherently make the control flow jump to execute whatever the malicious party wanted. That’s what bad guys started doing and which lead to DEP and ASRL. DEP stands for Data Execution Protection, a technology which separates the dynamic, software written data from the instructions that the CPU is going to execute. ASRL stands for Address-Space Layout Randomization, which changes the location of the software, the kernel code, the operating system’s libraries, and so forth, in the RAM so that the malicious party cannot know where certain pieces of software logic lies in the address-space which makes the use of known locations of useful code a no-go for execution. DEP and ASRL are both themselves really fascinating stuff and if you’re interested, go check them out in more detail, my few words here don’t really get you covered on them at all.

Return-Oriented Programming (ROP) is a way of getting control of the stack and changing bits and pieces of software logic towards the end of subroutines so that the CPU would then execute code that would diverge the program flow to an adversary route. That would of course, done correctly, lead to takeover of the operating system since the malicious party could execute whatever it wishes. Now Intel and Microsoft have come up with a new piece of fascinating technology called Control-Flow Enforcement Technology that adds more protection specifically to the stack operations. There are two new things to fight the problem:

1. ENDBRANCH instruction is added to the new x86/x64 architecture instructions sets. When the software is compiled with a CET supporting compiler that targets new CPUs, all legal (valid) calls and jumps are directed to an endbranch instruction. That is to say that whenever a subroutine is called, the first instruction to be found inside the subroutine must be an endbranch. If it’s not, the CPU throws an exception and the hell breaks loose. So the program gets compiled in a way that the subroutine/call/jump flow is enforced to obey the original intent of the programmer – thus the name Control-Flow Enforcement. Today’s processors can return to any valid place the kernel lets them to return to no matter what instructions lie there, but that’s going to be changed with CET. That makes it impossible for an attacker to redirect the program flow for example into middle of a library or so that could then lead to overtake. The brilliance of the endbranch instruction is that it’s implemented as a NOP in all the current Intel chips so it’s 100 % backwards compatible and requires no tricks from the software programmer.
2. Shadow stack which is a separate, hidden, you-cannot-touch-me stack living along the usual stack. For all the calls and jumps the software makes the return addresses are pushed to the stack but on the new chips they are also pushed to a new shadow stack – which happens automatically. Then when the pop op comes and it’s time for returning to a previous location, the CPU checks whether both the shadow stack and the normal stack have the same return address or not. If they match, ok, continue execution. If they don’t match, it means that someone has had their fingers on the normal stack’s return address and that all execution should be stopped. The programmer has no any sort of control over the shadow stack, it just operates in the background and is implemented in the hardware so that you cannot trick it nor touch it.

Together these two additions will make it absurdly hard to alter the execution flow.

If you found any of this interesting, do your homework for DEP, ASRL, CET and before that, take a look at Ars Techinca’s brilliant article How security flaws work: The buffer overflow. That’s a really nice piece on the whole stack and the security questions around it and gives you many pointers for more things to discover.

Some links: Intel’s blogposting, ROP on Wikipedia, TWIT.tv episode of Security Now

The post Control-flow Enforcement Technology to fight against ROP appeared first on Pietari Heino's personal website.