Work and research

High-frequency trading

I found a market niche in the high frequency trading space and created an autonomous trading system that has made >$300K profit so far on >$10M volume.

Due to the sensitive nature of this work, details are available upon request, but some highlights are:

• I wrote CUDA code that is 1000× faster than CPU SOTA for a particular set of math problems, drawing from textbooks, reference implementations, and math research papers. That code has been running continuously on 8 NVIDIA 3080s since 2023 Spring.
• I designed and implemented¹ a particularly tricky dynamic programming algorithm, reducing optimization times from seconds to tens of milliseconds. The algorithm is used for discovering optimal trading strategies, i.e. to determine how to arrange a set of trades, and at what priorities, to maximize revenue while minimizing fees and blocking rival trades.
• I generalized my trading strategies into a differentiable flow model, which I implemented in JAX and then bound to Rust via PyO3. This formulation discovers trades that were not previously found by prior heuristics, can be extended to non-market-neutral strategies, and is amenable to a variety of gradient optimization techniques.

Versions of the system were written in TypeScript, then OCaml, then Rust with an OCaml sidecar. The whole system runs on a geographically-distributed Kubernetes cluster on a low-cost cloud provider².

Google Research Mountain View, CA

In Google Research, I worked on AI research and infrastructure under Philip Nelson, Kevin Murphy, and Sergey Ioffe.

In that time I hosted 3 interns:

• Samuel Yang, who later joined Research.
• Andre Esteva, who has since been published multiple times in Nature and is the founder of an AI medical diagnostics company.
• Liam Fedus, who later joined Research and then joined OpenAI to help create ChatGPT.

I also:

• Interviewed 100s of candidates.
• Earned the “expert” designation in Google’s internal StackExchange clone (YAQS) for answering questions tagged #tensorflow, #python, #tpu, and #spanner.
• Got readability for Golang, C++, and Python.
• Finished my PhD.

Note, I quit six months before the first layoffs in 2023.

Here are some selected projects:

In Silico Labeling

In Silico Labeling was a project that used deep learning to predict fluorescence images from transmitted-light images of unlabeled cells. It gives life scientists many of the benefits of fluorescence labeling without most of the costs; see this blog post and this editorial for context.

I originated the idea and led the effort across an 18-person team at Google, Verily, Harvard, and Gladstone. The work consisted of target identification, experimental design, sample creation, data collection using robotic microscopes, large scale distributed image processing, and model development.

At the time, the SOTA for image-to-image models wasn’t good enough, due to limited spatial context, artifacts caused by scale changes, and convolution edge effects. So, I created a new architecture (second two figures) carefully designed to address these issues, resulting in a 25% loss reduction and qualitatively better images.

This work was published in Cell, the the Google Blog, and open sourced. It was also patented and led to the creation of two new projects at Verily. Later work automated quality control in similar pipelines.

For this project I used C++, Golang, Python, Flume, and TensorFlow.

Hyperparameter tuning

I created Google's first hyperparameter tuning API for deep learning, by providing a convenient interface to black box optimizers and infrastructure to manage experiment lifecycles. This was the first version of what became the Vertex AI hyperparameter tuner, a product of the Vizier team led by Daniel Golovin.

At the time, the Vizier team already provided black-box optimizers for other Google products (e.g. Ads), but the API was not immediately suited to deep learning. I created a service and an API the user could use to define a search space along with hooks into their training and evaluation code. My infrastructure then ran the optimization, including selecting the next experiment, scheduling it, collecting evaluation data, dealing with failures, etc.

Fun fact: At the time I became the biggest user of Brain compute at Google, as I used the system to tune the hyperparameters of my own models. This compute was all low-priority "free" compute obtained by migrating my jobs around the globe to follow the night, taking advantage of overcapacity.

For this project I used C++, Python, and of course GCL³.

Example selection

I built a system that trains deep networks faster by dynamically adjusting the train set data distribution (cf. curriculum learning), providing a nearly free 30% training speedup on tasks with imbalances in example difficulty, such as image classification.

The main idea was to reduce the variance of the SGD gradient estimate via importance sampling. The importance weights were estimated on the fly via a concurrently-trained helper network, using current model parameters. Interestingly, the curricula produced by the system were often human-interpretable and provided insight into the task.

Other than the proper design of the helper network, the main difficulty was to make the system fast, as it needed to feed TPUs without bottlenecking them⁴. The final artifact was a distributed system consisting of data loaders, annotators, the caching sampler, and the concurrently-trained helper network, all communicating via a cluster-local DB, achieved in about 50K lines of C++, Python, SQL, and GCL.

Unfortunately, at the time TPUs were plentiful and the extra lifting required to integrate the system was not seen as worth it, causing it to be deprioritized.

Miscellaneous

• I published on neural architecture search and model calibration and ensembling.
• I created an internal JAX and TensorFlow library to automate some of the work in model calibration and ensembling. That library was used to create the motion blur model on Pixel devices.
• I ported the internal TensorFlow library based on this paper to JAX, added features requested by clients, and maintained it.
• I wrote LabeledTensor with Stephan Hoyer, a TensorFlow library for endowing tensors with semantically meaningful dimension and coordinate labels. Note the author attributions on GitHub are incorrect here, due to the way in which internal code was synced to GitHub; I wrote about half of the code.

Willow Garage Menlo Park, CA

In 2012 June I started a 3-month research internship at Willow Garage, working in robot perception. I liked it so much that I twice extended the internship, finally ending in 2013 March.

While there, I:

• Developed a similarity-invariant version of the LUCID descriptor, called eLUCID, which was especially fast on mobile devices (C++).
• Developed a similarity-covariant local descriptor. This was my first exposure to large-scale compute, as I used the Willow Garage cluster to optimize the descriptor parameters (Figure 4) (Scala and C++).
• Created Billy Pilgrim, an open-source evaluation framework for local descriptors, intended to replace VLBenchmarks. It is broken into a backend (Billy) and a frontend (Pilgrim). The name is an unfortunate reference to Slaughterhouse-Five (Scala and C++).
• With Andrey Pavlenko and Andrey Kamaev of Itseez⁵, I added Java to the list of supported languages for OpenCV by modifying the interface generator they used for other languages. I did it so I could use Scala⁶ for research 😉 (Java, C++, lots of CMake⁷).

I also organized two programs for the wider benefit of the company:

• I taught a twice-weekly CrossFit class, using equipment Willow purchased for the purpose.
• When Willow had to lay off its kitchen staff, I organized company-wide catering, paid for by the employees. This inspired me to create Food for Thought at UCSD (see Education).

Google Mountain View and Los Angeles, CA

• In 2011 I was at the LA office, working on Google Goggles research and backend infrastructure, including adding the first high-dimension log-time nearest-neighbor method to the scalable matching service (C++ and Python).
• In 2010 I was at the Mountain View office, where I helped the webcrawler to detect and appropriately handle auto-generated websites (C++).

Education

University of California, San Diego La Jolla, CA

I got a PhD in computer science, with foci in computer vision and machine learning, publishing in ML theory, computer vision (link, link, link, link, link, link) , and deep learning.

My initial focus was machine learning, working with Charles Elkan. During this time, I attended the Machine Learning Summer School at Cambridge University, where I presented a paper on theoretical machine learning.

In 2009, I switched my focus to computer vision with Serge Belongie. My initial thesis area was in local descriptor methods, e.g. SIFT, which are used in computer vision to compare local regions of images. They are building blocks for many computer vision applications, including structure-from-motion and object detection and recognition.

Between 2010 and 2013, I did a total of three internships at Willow Garage and Google amounting to 16 months.

Inspired by the internships, I developed the Food for Thought (FFT)⁸ program, paid for with grants and with the support of my advisor Serge. FFT provided Google-style free food to all members of our lab at UCSD. I believe it significantly improved lab morale and communication. Here's a photo of some of the lab eating and here's one of the stocked fridge.

I also did some teaching:

• 2014: TA for CSE 202, graduate algorithms (UCSD)
• 2013: Google Summer of Code mentor for OpenCV
• 2013: TA for CSE 255, data mining (UCSD)
• 2010: TA for CSE 252B, graduate computer vision (UCSD)
• 2010: TA for CSE 202, graduate algorithms (UCSD)
• 2009: TA for CSE 105, undergraduate computability (UCSD)

In 2014 I joined Google Research and continued my PhD part-time, while pivoting to focus on deep learning applications in computer vision. I finished in 2018.

Swarthmore College Swarthmore, PA

While an undergrad at Swarthmore College, I worked in the summers with Gary Cottrell on a variety of cognitive science topics, thanks to whom I developed an interest in computer vision and biologically inspired models.

In 2008 I graduated with honors with a BA in math and a minor in computer science.

Code projects

During my PhD, I kept sane by working on a number of side-projects, for example:

• PersistentMap: A type-safe, boilerplate-free, key-value store for Scala.
• salve: A macro and template library for adding some functional programming ideas to C++.
• sbt-latex: A build management tool for LaTeX (Scala).
• CharikarLSH: An implementation of Moses Charikar's method for approximate nearest neighbor retrieval. Note, techniques like this are how vector databases work (C++).
• mbtree: An implementation of metric-ball trees for nearest neighbor search (Scala).
• DistanceLSH: An implementation of a metric hashing for nearest neighbor search (Haskell).

Publications

• X. Wang, D. Kondratyuk, E. Christiansen, K. M. Kitani, Y. Movshovitz-Attias, and E. Eban, “Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
• C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, “NAS-Bench-101: Towards reproducible neural architecture search,” in International Conference on Machine Learning, 2019, pp. 7105–7114.
• S. J. Yang, M. Berndl, D. M. Ando, M. Barch, A. Narayanaswamy, E. Christiansen, S. Hoyer, C. Roat, J. Hung, C. T. Rueden, A. Shankar, S. Finkbeiner, and P. Nelson, “Assessing microscope image focus quality with deep learning,” BMC Bioinformatics, vol. 19, no. 1, pp. 77:1–77:9, 2018.
• E. Christiansen, S. J. Yang, D. M. Ando, A. Javaherian, G. Skibinski, S. Lipnick, E. Mount, A. O’Neil, K. Shah, A. K. Lee, P. Goyal, W. Fedus, R. Poplin, A. Esteva, M. Berndl, L. L. Rubin, P. Nelson, and S. Finkbeiner, “In silico labeling: Predicting fluorescent labels in unlabeled images,” Cell, vol. 173, no. 3, pp. 792–803.e19, 2018.
• P. C. Nelson, E. Christiansen, M. Berndl, and M. Frumkin, “Processing cell images using neural networks,” May 15 2018, US Patent 9,971,966.
• E. Christiansen, “From local descriptors to in silico labeling,” Ph.D. dissertation, University of California, San Diego, 2018.
• E. Christiansen, V. Rabaud, A. Ziegler, D. Kriegman, and S. Belongie, “Match-time covariance for descriptors,” in British Machine Vision Conference, BMVC 2013, Bristol, UK, September 9-13, 2013, 01 2013, pp. 12.1–12.11.
• E. Christiansen, I. S. Kwak, S. Belongie, and D. Kriegman, “Face box shape and verification,” in Advances in Visual Computing, G. Bebis, R. Boyle, B. Parvin, D. Koracin, B. Li, F. Porikli, V. Zordan, J. Klosowski, S. Coquillart, X. Luo, M. Chen, and D. Gotz, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 550–561.
• A. Flores, E. Christiansen, D. J. Kriegman, and S. J. Belongie, “Camera distance from face images,” in Advances in Visual Computing - 9th International Symposium, ISVC 2013, Rethymnon, Crete, Greece, July 29-31, 2013. Proceedings, Part II, ser. Lecture Notes in Computer Science, G. Bebis, R. Boyle, B. Parvin, D. Koracin, B. Li, F. Porikli, V. B. Zordan, J. T. Klosowski, S. Coquillart, X. Luo, M. Chen, and D. Gotz, Eds., vol. 8034. Springer, 2013, pp. 513–522.
• A. Ziegler, E. Christiansen, D. J. Kriegman, and S. J. Belongie, “Locally Uniform Comparison Image Descriptor,” in Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, P. L. Bartlett, F. C. N. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., 2012, pp. 1–9.
• T. Winlock, E. Christiansen, and S. J. Belongie, “Toward real-time grocery detection for the visually impaired,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2010, San Francisco, CA, USA, 13-18 June, 2010. IEEE Computer Society, 2010, pp. 49–56.
• E. Christiansen, “An upper bound on prototype set size for condensed nearest neighbor,” arXiv preprint arXiv:1309.7676, 2013.
  • I actually wrote this in 2009, when I was a new grad student and still had a pretty strong math bent.
• J. P. McCleery, L. Zhang, L. Ge, Z. Wang, E. Christiansen, K. Lee, and G. W. Cottrell, “The roles of visual expertise and visual input in the face inversion effect: Behavioral and neurocomputational evidence,” Vision Research, vol. 48, no. 5, pp. 703–715, 2008.
• M. H. Tong, A. D. Bickett, E. Christiansen, and G. W. Cottrell, “Learning grammatical structure with echo state networks,” Neural Networks, vol. 20, no. 3, pp. 424–432, 2007.

Miscellaneous

Footnotes

1. With Lance Hepler, a friend who joined for 3 months between jobs.

2. This low-cost cloud provider was the origin for one of the most annoying bugs I’ve recently experienced, related to VLAN MTU mismatches and randomly dropped packets.

3. The Generic Config Language, a Google-internal language for deploying and configuring services.

4. I believe the bus was around 12.5 GB/s at the time or 83K ImageNet examples / second.

5. At the time an OpenCV core contributor, later acquired by Intel.

6. I’ve had an interest in functional programming ever since I took Ranjit Jhala’s programming languages class at UCSD. I am so happy Rust is now making these ideas mainstream.

7. The hardest part was augmenting the build definition, which at the time was was hundreds of pages of CMake files with all kinds of terrible interdependencies, global mutable state, and of course no types. To make sense of it I printed it all out and spread it over a large conference table (it took all the space several pages deep), and kept spatially re-arranging and marking it up until it made sense.
8. The website itself is now offline.