Joining Forces with Augury

Today is an extremely exciting day for Alluvium. After many months of conversations, experimentation, and now integration, I could not be happier to announce that Alluvium has been acquired by Augury.

What this merger means for our ability to deliver value to industrial customers is immense. But, before we look ahead, it is worth looking back in order to fully appreciate the potential of this opportunity. When we founded Alluvium our mission was to leverage all available operational data industrial systems were already generating, and deliver immediate and high-value insights in real-time to operators. One of the things that always differentiated Alluvium is our technology. Specifically, our ability to ingest any kind of operational data, learn from it using state of the art techniques, and turn that learning into accessible and meaningful insight. What made Alluvium special as a company, however, were the people and our approach to delivering that technology.

We knew from the beginning that empathy was key:

In all data there is humanity. In every bit there are traces of this humanity: in how a choice is made, or how a system is built. In the physical world the complexities of this humanity are magnified. To manage this complexity requires both deep technical expertise and innovative engineering. It also requires considerable empathy for the human beings behind that data.

The Alluvium and Augury teams have known each other for a long time. Along with consistently showing up on the same lists and logo boards, we also once even worked out of the same office space. Through that proximity I had the chance to meet Saar and Gal – Augury’s co-founders –  a few times. Beyond the success of their business, what always impressed me was their clear articulation of vision, and their ability to precisely execute on that vision. And though our respective visions overlapped, we came at the problems from entirely different directions.

Augury has developed a bottoms-up hardware and machine learning stack to continuously monitor industrial equipment, starting with specific equipment types and growing from there. Alluvium came at the problem from the opposite direction, developing a unique software platform for ingesting all operational data to provide immediate top-down value from data customers already had. By combining these two approaches, and the underlying technologies, we will greatly expand the landscape of customers and business problems we can address. It is truly a phase shifting opportunity for industrial analytics.

Finally, while Alluvium and Augury started with different technology and product approaches, there is one consistent similarity between the two companies: empathy. It is our shared values, both for how to work with each other and customers, that has stood out above all through this process. We are excited for what we will build together, but more importantly, we are excited for the future we will be able to deliver to the men and women working in complex and dangerous industrial operations. You can read Saar’s own thoughts on this opportunity.

The mission continues.

‘Building Data Science Teams’ on the DataFramed Podcast

Recently, I sat down with Hugo Bowne-Anderson for his DataFramed podcast. We had a lively conversation, and covered many topics, including the challenges of building data products for industrial customers, and our approach to recruiting at Alluvium.

If you would like to learn even more about our recruiting process, we would love to hear from you.

 

Look back, plan ahead

On August 23, 2017 Alluvium turned two years old. When I think back to founding the company those 29 months ago, it is truly a distant memory. This is primarily because of how much we have learned and built since then. Yet, unless you follow me on social media, or happen to have gone to one of a handful of meetups or conferences I attended, it would be difficult to know much about what we have learned and built. Today – I am very happy to report – that has changed.

Today we are re-launching our website, and in doing so finally bringing to light the hard work of our team. Through their collective efforts, and with the help of the incredibly creative and thoughtful team at Little & Co., I am extremely proud of our new public face. I encourage you to look around, but I also want to take some time to discuss some of the most important learnings we have collected over the past two-plus years.

Stability Score

There is a platitude in building data products the goes something like, “Never use machine learning for machine learning’s sake.” This sentiment was brilliantly elaborated in Google’s now seminal paper, Machine Learning: The High Priced Credit Card of Technical Debt. Yet, despite deep recognition of this problem, many people still ignore it. I am sympathetic. It is fun to experiment with new methods and then imagine a product and use case to fit.

Alluvium’s mission has always been to build tools that support men and women working in complex industrial settings. One of the first things we learned as we began to meet and observe these men and women in their work, and in particular observe how they interact with data and use it to make decisions, is that a unique tension exists between data discovery and data reasoning.

Take, for example, the data generated by a typical process control system. Most of the time that data looks identical from one moment to the next, and when things change, the change is consistently predictable. The challenge lies in discovering meaningful changes in these data without dedicating a tremendous amount of operator focus, which is arguably the most valuable resource in a manufacturing environment. Moreover, discovering these changes is not the core competency of the operators and engineers working with process control systems. Their training and experience equips them to reason about these data, but because this tension is pervasive, the potential value of that data is greatly diminished. Now, scale that problem up from a single control system to an entire plant, and from an entire plant to an organization’s global operation.

There are many potential ways that artificial intelligence can be used to begin addressing this problem, but this can also be a trap. Throwing machine learning techniques at this data creates a maze of product paths that can swallow a startup. Critically, we sought to seek firsthand knowledge, and after listening to operators and observing their workflows, it became clear the best application was dimensionality reduction. In moving the cognitive load of data discovery to an AI, we can relieve some of this tension and allow users to focus on reasoning about data, rather than working to discover novelty. This is the essence of Alluvium’s Stability Score, which is at the core of all our work. We have taken both a deliberate and opinionated approach to applying machine learning to industrial operations because we believe reducing the complexity of what operators must observe to be able to reason about their work is the most the best way we can support their work.

Alluvium Primer

Soon after Alluvium was founded, I penned a short post describing my motivations for starting the company. A central focus of that post is the challenge of dealing with data from the physical world, and the unique opportunity this presented for leveraging human expertise in the data generating process to build novel data products. Our early intuition was to build products that could provide relief at the earliest point in the data discovery process. The term of art for this in industrial operations is “edge analytics,” analyzing data in real-time as production systems are in operation to provide the most timely insights.

In pursuing this intuition, we endeavored to build a core software platform for performing real-time Stability Score analysis from industrial data streams. As the principle engineering moved forward and we began to look for early proof-of-concept (POC) partners, we discovered an uncomfortable truth about the industrial market. While many potential partners were sold on the value, and even aspired to use sophisticated artificial intelligence tools in their production systems, the institutional barriers to actually accomplishing this were steep. The very idea of integrating a new tool into the critical workflow of an industrial operations was a tall order that was further complicated by the fact that we were a startup no one had ever heard of using techniques completely foreign to the industry. Not surprisingly, our initial efforts to build POCs at “the edge” fell flat.

While these early efforts were largely unsuccessful, we observed a distinct pattern. Potential partners would note that while edge deployment of a POC inside a production center was challenging, they were still eager to test the technology. In fact, we heard a similar story from several early customers. For years, many of them had been collecting and storing massive troves of production data that could be used to “replay” a production scenario as a POC. We started with a single one-off POC to support this discovery-through-replay paradigm, and eventually it became clear that we had discovered the market’s point of adoption. We found ourselves well-positioned to build a product to support it.

That realization became our flagship product: Alluvium Primer. I could not be prouder of the work the team has done in building Primer, and more thankful to our early pilot customers for their patience and conviction.

The Team

I do not like using growth as a metric for company success, particularly in the early days. That said, our team has more than doubled in the last year, and I am honored to work with such a deeply talented team. The breadth of this talent, and diversity of their collective life experiences, speaks to the unique challenges of our customers and the strength of our shared vision.

If you have made it this far, you may be curious as to how to join our team. I am happy to say that we are looking to fill many open positions, and to continue to increase our team’s diversity and the depth of our talents.

This Blog

What excites me most about our website relaunch is this blog. I am eager to getting back to writing, and in particular writing about the education and experience of building Alluvium. I am also eager for my colleagues to have a chance to share their ideas and experiences, from the technical to the personal.

I will be starting with a topic that I am often asked about: recruiting. I suspect that everything I have to say on the topic will not fit into a single post, but be sure to check back soon.

A view from the start

Every new company has an official “first day.” In most cases, this is only meaningful to lawyers and HR/payroll systems. Most often, founders have been thinking and building for weeks, months, even years before that official first day. Despite its potential ephemeral nature, however, the day still represents a milestone.

When Alluvium’s first day was finally set on the calendar I did not want it to pass without notice. In fact, I wanted the day to be special — both memorable and inspiring. Being fortunate enough to live and work in the greatest city in the world, there were many potential locations. Ultimately, however, the decision was easy. There was only one place that might capture the feeling of that moment, and provide a lasting impression of the grandness of our endeavor: the top of One World Trade Center.

Early on a hot and humid NYC summer morning I met the team on the observation deck. As we circled the — truly — spectacular views of our city, we talked. Mostly, about “why”. Why were we starting this company; why were we the right people to start it; and, why should Alluvium even exist? We spent the most time on this last point, as it was something I had spent a lot of time thinking about leading up to the first day. In those preceding days I had scribbled out pages of notes thinking about ways to answer that question. On one sheet I drew two intersecting circles that read “data” and “humanity”, respectively, with Alluvium written in the intersection (yes, a Venn diagram).

It is easy to fall in love with the technical details of building a complex software system. I find this to be particularly pernicious in the Big Deep Data Science Machine Intelligence industry. I too am often swept away in these circular discussions. What is much harder, however, is falling in love with the entrenched human problems at the core of what these systems are meant to address. This is what made our view from the observation deck of One World Trade Center the perfect place to start. In every direction we could see the scale of human complexity. And, perhaps in the starkest of terms, were forced to think about the challenges that we would face by focusing on building a system that seeks to support human operators, rather than replace them.

Looking down on the city, which at that height is revealed much more as a living organism than a planned landscape, I remember telling the team that if there was ever the possibility that a single technology could empower every single person down there; living and working inside one of the most complex systems in the world, than that is what we are building. That is why we exist, so remember this view.

I am sharing this story now because Alluvium recently celebrated another milestone: our first anniversary. In this year we have learned and built some amazing things. While we have not shared much of that, I look forward to sharing much more about what we are building, and for whom, over the next several weeks. Over this time we have also grown, a bit. As part of that growth I have taken to having all new employees meet me at the observation deck on their first day. We talk about why Alluvium exists, and why they are the right person to continue what we started a year ago. Our first company tradition.

I have now done it several times, and beyond the awesomeness of the spectacle, I love having the opportunity to be re-humbled by the view, and see the reaction of my new colleague. Though it is “touristy,” I recommend visiting One World Trade Center to everyone who will listen — even New Yorkers. But that goes double for entrepreneurs, and those intrepid enough to join them.

But, if you cannot make it to New York, I hope you can find your own humbling view. One that unveils a tapestry of opportunity, complexity, and hope. A view from the start.

Alluvium

In these waning days of summer it is common to reflect on how the days were passed. For me, it has been a wild few months. I spent these hot and sticky New York City days building the vision, team, and financing for a new company. I am incredibly excited, and honored, to report that last week was our first “official” week of work.

We are Alluvium, a team of engineers, data scientists, and executives building deeply integrated tools and services to deliver value for industries facing data challenges in the physical world. We believe that data emitted in the physical world is unruly and unrealized, but holds the keys to today’s largest business challenges.

Our mission is to build products that address these challenges, and to give the people working in these industries enhanced abilities. The journey has begun; but, how did we get here, and where are we going?

How did we get here?

The industry of data, or “big data” if you prefer, is young. But, not so young that it is without perspective. I have spent my whole career — now over a decade — working in data, of all sizes. The “big data” industry started in earnest around the mid-2000’s with the development of a few seminal technologies that provided useful abstractions for both distributed data storage and computation. These technologies were developed primarily to support improved web search, and their historical origins had a large influence on how the ecosystem developed.

The technology was developed by and for products generating data in the digital world, and the first generation of the industry focused on building products to solve those problems.

Problems in the digital world, i.e., first generation of the big data industry

An ecosystem of tools, services, and companies have been built to address these digital problems. This is by no means meant to downplay those contributions. A decade later, and we built some amazing technology and products. These are, for the most part, solved problems. What remains unsolved are data problems in the real, physical, world.

The next decade of the big data industry will be about solving these problems. Borrowing what we know about building highly available, scalable, smart systems, and inventing new systems for analyzing streams of data emitted when analog actions and decisions occur.

This is both a natural progression of the industry, but also a fundamental shift in the kinds of technologies, people, and companies that will constitute the next generation of the data industry.

Where are we going?

The promise of better living through connected devices, the so-called “Internet of Things,’’ has captured the popular zeitgeist. While the entire consumer electronics industry may be set to instrument the lives of consumers — from wearables to smart homes — much of the focus remains on designing and marketing devices that engage consumers over a long period time.

All of the attention paid to imagining this whimsical future belies the present reality: there are massive industries emitting countless streams of data today that are remarkably underutilized.

Problems in the physical world, i.e., the next generation of the big data industry

The next frontier is not about making comfortable lives better through connected devices. It is about building data-driven products that make the hard, dangerous, and crucial jobs that power the global economy frictionless, safer, and more reliable.

In all data there is humanity. In every bit there are traces of of this humanity: in how a choice is made, or how a system is built. In the physical world the complexities of this humanity are magnified. To manage this complexity requires both deep technical expertise and innovative engineering. It also requires considerable empathy for the human beings behind that data.

Those of us who work with data are fond of describing it as messy, but data from the physical world is more than simply messy. It is knotted up in the perpetually flawed mechanism used to convert analog actions to digital signal, and the humanity that underlies it. The complexity of this humanity, however, is also our greatest strength and opportunity. The expertise, experience, and bias that people imprint on the data provide material for building great products.

This is where our journey begins.

Where are you?

We have started with a team that I am extremely proud to call my colleagues and partners. They have built some of the most used and recognized products in both consumer and enterprise data analytics. The combined expertise already under-the-hood here at Alluvium is intimidating in the best possible way.

But, hard problems require many more smart people to solve them.

We are in the very early days, but if this scale of opportunity and challenge is something that excites you, and makes you want to jump out of your chair and start building, we want to meet you.

Send along a note to us at info@alluvium.io, or fill out this short form to tell us a bit more about yourself.

It is time to solve the next generation of really hard problems.

Thanks to Chris B.

Alluvium has been acquired by Augury to provide unprecedented transparency into the health of your operation.