Successful businesses make better decisions than their competitors. Competition is increasingly fierce and the marketplace unforgiving of organizations that don&rsquo;t execute well.
A/B testing has become a common technique to improve product experience, but often A/B testing is poorly implemented, leading to a facade of &ldquo;good&rdquo; test results masking poor business performance. You may be familiar with this effect, your team runs many tests, some of them succeed, but the overall business performance does not improve. Somehow 2% + 10% + 3% + 7% + 15% = 0% or 1%.
Common causes of this are
<ul style="margin-left: 20px; margin-top: -10px;">
<li>Bad data</li>
<li>P hacking</li>
<li>Peeking</li>
<li>Optimizing for the wrong thing</li>
</ul>
<h3 style="margin-bottom: 6px;">Bad data</h3>
Bad data ruins everything. It doesn&rsquo;t matter how sophisticated your statistics and analytics are if they are built on a foundation of bad data. And producing good data is&nbsp;hard. Bugs, often subtle, lurk throughout the data pipeline: from front end clients producing incorrect data, to inconsistent data contracts between systems, to ETLs that misprocess the data, to dashboards with the wrong logic.
&nbsp;There are few general approaches to ensuring you have good data:
<ul style="margin-left: 20px; margin-top: -10px;">
<li>Use battle tested systems with bullet proof implementations</li>
<li>Use declarative, rather than procedural approaches.</li>
<li>Test your data.&nbsp;
<ul style="margin-left: 20px; list-style-type: circle;">
<li>Run known data from the edges of your system all the way through and make sure everything lines up as expected.&nbsp;</li>
<li>Reconcile your data. Look at it from multiple &ldquo;angles&rdquo; and makes sure it is consistent</li>
</ul>
</li>
</ul>
Good data is hard, and bad data can be hard to spot. It takes effort to consistently produce reliable data. Don&rsquo;t neglect it.
<h3 style="margin-bottom: 6px;">P-hacking</h3>
<a href="https://en.wikipedia.org/wiki/Data_dredging">P-hacking</a>&nbsp; &ldquo;is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. This is done by performing many statistical tests on the data and only reporting those that come back with significant results.&rdquo;
Obligatory <a target="_blank" href="https://www.explainxkcd.com/wiki/index.php/882:_Significant" rel="noopener">XKCD comic</a>.
P-hacking is a well known phenomenon, but the incentives for it are strong, so without controls it tends to happen. The typical story arc is as follows: a new treatment is developed and tested and it comes back flat on the primary metric, say revenue per user. But the feature is cool, and everybody wants a win, and it turns out it is up on clicks per user, so that&rsquo;s something! Write up a deck and tout how engagement was increased.&nbsp;
But sadly, it is unlikely that there was a real win.
Preventing p-hacking requires organizational discipline, and software can help. Software should require that experiments register a primary metric on which the experiment will be judged. Alternatively, software can adjust confidence intervals to account for multiple comparisons.&nbsp;
Registering a primary metric is the better approach though, as there should be a hypothesis on what the experiment will achieve, why, and how that will drive the primary metric. Adjusting for multiple comparisons will usually reduce confidence to the point that nothing will be considered a winner.
Going back to the story arc, if data spelunking shows that conversion is way up, and that is an interesting business goal, the correct thing to do is to run the experiment again with a new primary metric. This will verify or refute the hypothesis that treatment improved conversion.
<h3 style="margin-bottom: 6px;">Peeking</h3>
A t-test is a commonly used statistical test to determine if an experiment is successful. The catch is that you can not peek at the results. That is you have to let the test run for a predetermined amount of time and only then look at the results.
As with p-hacking, peeking is a well known phenomenon and the incentives run the wrong way. People like to peek, and people like to roll out a test that is up, even if the experiment is not over.
Unlike p-hacking it is fairly easy to enforce correctness. Either use some other type of statistic that doesn't suffer from peeking, like sequential or bayesian stats, or use software that does not show you the value of a t-test until the experiment is over. If you choose the latter though, make sure to have guardrails in place in case the test is catastrophically down.
<h3 style="margin-bottom: 6px;">Optimizing for the wrong thing</h3>
It is often the case that a business goal is hard to measure and so a proxy is used instead. For example, if the sales cycle is long, measuring the impact to revenue of a change on a landing page will be noisy and hard. As a result, it is tempting to measure landing page conversion instead. But improvements to landing page conversion may not improve revenue. You might just be pushing more low quality traffic down the funnel.
Sometimes measurements don&rsquo;t measure what&rsquo;s intended. For example, with search, a click on a result could naively be interpreted as a success. But it might not be! The user may see the page, immediately decide it&rsquo;s not what they want, and return back to the search results page. To make this measurement better, you need to account for search bounces.
In all cases it is important to ensure that you are moving business metrics that matter. A good way to do this is to use causal inference. Causal inference will tell you if the metric you are experimenting with is actually impacting business outcomes. Running A/B tests where you directly measure a business metric like revenue is obviously one great way to determine a causal relationship. Where that is hard to do, you can use techniques like two-stage least squares to determine if your proxy metrics actually matter to your business.
<h3 style="margin-bottom: 6px;">Summary</h3>
A/B testing is a valuable tool, but it&rsquo;s easy to misuse, giving an illusion of progress without real tangible benefits. Ensuring the integrity of data, understanding the nuances of statistical tests, and setting correct metrics for evaluation are crucial steps in leveraging A/B testing.
If you found this interesting, and are looking to run effective A/B tests, please take a look at Causal. We have fantastic tools that help you successfully optimize your products. We&rsquo;ve deeply thought through the challenges associated with optimizing, and have an integrated approach that is unmatched in the industry.

Successful businesses make better decisions than their competitors. Competition is increasingly fierce and the marketplace unforgiving of or

How to A/B test

Unlock the true potential of A/B testing with the right techniques and tools. Avoid common pitfalls and ensure genuine business growth.

<h1>Product development is a mess</h1>
Imagine writing an API, but without any formal specification. No OpenAPI, no GraphQL, no other technology to coordinate the client and server. How well would that go? It would be a disaster.

So why do many organizations fail to use any formal specification when coordinating systems more complicated than an API call?

It is extremely common to have disparate systems, owned by different teams, coordinating to support product development. There are front end teams, api teams, service teams, ML teams, all trying to coordinate. Numerous challenges inevitably arise.

Teams are coordinating and supporting:
<ul style="margin-top: -8px;">
<li style="margin-left: 20px;">Data Analytics</li>
<li style="margin-left: 20px;">A/B Testing</li>
<li style="margin-left: 20px;">Recommender systems</li>
<li style="margin-left: 20px;">ML model training and support</li>
<li style="margin-left: 20px;">Outcome evaluation</li>
</ul>

And they run into the following challenges:
<ul style="margin-top: -8px;">
<li style="margin-left: 20px;">Application events are often incorrectly coded</li>
<li style="margin-left: 20px;">ETLs are time consuming to write, prone to errors, and difficult to verify</li>
<li style="margin-left: 20px;">The data is not in an easy to use format for the tasks at hand</li>
<li style="margin-left: 20px;">Data that should have been captured, wasn&rsquo;t</li>
<li style="margin-left: 20px;">Multiple functions or teams need to be involved for seemingly simple changes</li>
<li style="margin-left: 20px;">It&rsquo;s difficult to evaluate if optimizations are actually moving the business forward</li>
</ul>

In short, everything is harder and more error prone than it needs to be.

Not only does the lack of a specification lead to chaos in development, but it also has strategic implications - risking the overall product vision and long-term scalability.
<h1>Data aware product development to the rescue</h1>
So, why not use the same solution for product development that is used for APIs? That is, create a contract between components that specifies intent. Specifying intent is helpful for people, and allows tools to understand objectives.
Data aware product development is akin to the structured framework provided by API specifications like OpenAPI and GraphQL, but applied to the broader context of product development. It establishes a clear agreement on how to collect and evaluate user interactions. In doing so, it ensures smoother collaborations, and enables tools to do much more heavy lifting. It streamlines processes, reduces errors, and unlocks new capabilities.
Just like a construction team, a product engineering team should insist on a blueprint. Data aware product development is the solution for contemporary product engineering teams.
<h2>Optimizing a product</h2>
To optimize a product, tracking user interactions is vitally important. As a concrete example, take an ecommerce cross sell. For each cross sell view, the following should be tracked:
<ul style="margin-top: -8px;">
<li style="margin-left: 20px;">What was the user&rsquo;s previous page view history and items in cart</li>
<li style="margin-left: 20px;">What items were recommended</li>
<li style="margin-left: 20px;">What items did the user click on</li>
</ul>
More generally, for any interaction, the following should be tracked:
<ul style="margin-top: -8px;">
<li style="margin-left: 20px;">What is known about the user at the start of the interaction</li>
<li style="margin-left: 20px;">What the user saw</li>
<li style="margin-left: 20px;">How the user responded</li>
</ul>
The heart of data aware product development is to define contracts that represent the above pattern and to utilize tools that understand these contracts.
<h2>Contracts</h2>
Contracts are structured documents that clearly define the type, format, and relationship of data elements required for a specific product or service. Think of them as blueprints for data.
By establishing a consistent set of guidelines, contracts minimize ambiguities and discrepancies, ensuring all teams and all tools work from a unified point of reference.
Because of their structured nature, contracts facilitate automation. Tools know what data to expect, in what format, and how to process it.
Contracts are adaptable. As business needs evolve, contracts can be updated to reflect new data requirements, ensuring that systems can easily adapt.
<h2>Tooling</h2>
Once contracts are defined, tooling:
<ul>
<li>
Ensures clients collect correct data A client library is generated from the contract. This library is strongly typed and ensures the collection of all relevant data, the correctness of all types, and adherence to a recognized format.
</li>
<li>
Generates and runs complex ETLs Collected data is automatically processed, stitched together, and summarized. Data associations are defined in the contract, and since the gathered data adheres to a recognized format, the production of summary output tables is well defined. Within the ETL, custom metrics can be created, events can be linked to impressions, durations can be computed, and a lot more, all without the need for manual ETL engineering.
</li>
<li>
Enables updating applications w/o additional engineering work Data outputs are added to the generated client library, which in turn makes the data available to applications. The data can range from simple text, to ML model selection. Given that the client offers a clearly defined interface, changes to the data are entirely transparent to applications. A/B tests can be conducted, rollouts can be configured, and the UI can be updated without the need for extra engineering tasks.
</li>
<li>
Creates training sets for ML models ML models usually rely on data points consisting of ML features, model outputs, and associated payoffs. The contract outlines these components so the client can guarantee accurate data collection and the ETL can automatically generate tables for model training in the optimal format for data scientists.
</li>
<li>
Evaluates outcomes A/B tests are effortlessly run. Models are trained and deployed. Relevant metrics and analytics are automatically generated. Easily see how your work affects your business.
</li>
</ul>
<h1>Summary</h1>
Just as APIs rely on a specification to function properly, the same principle applies to larger systems. Data Aware Product Development offers a systematic approach to handling data needed for product development.
By establishing flexible contracts, all teams and all tools work from a unified point of view, increasing efficiency and delivering a raft of capabilities that otherwise would be error prone and time consuming to implement and manage.
Are you a technical leader looking to dramatically improve your product development strategy? Dive deeper into data-aware product development with Causal and witness the benefits firsthand.

Unlock efficient product development with data aware strategies: streamline processes, reduce errors & enhance collaboration through data contracts

Data Aware Product Development

Build product like you’d build any other software component

Learning to Rank Initially, when you put together your search solution, you’ll tweak different parameters to have more relevant answers to y

Learning To Rank with Causal

Simplify your real time machine learning development process and improve search relevance by using Causal

Improve your search with learning to rank

<h2>Overview</h2>
We wrote this article for teams experimenting with chatbots as part of their product. We&rsquo;re going to share potential goals, considerations, challenges, and measurement tips to help you effectively test the impact that a new chatbot may have on your business and user experience.
<h2>Goals: Why add a chatbot to your product?</h2>
As with any new feature, your goals are likely to make your business perform better and your users more satisfied with their experience.&nbsp;
To know your &ldquo;why&rdquo;, consider your user audience and use case. Are your users going to be comfortable using chat? &nbsp;Does the user goal you&rsquo;re trying to address lend itself to natural language prompts?&nbsp; Let&rsquo;s consider the overall impact a chatbot has on user experience. Your users have likely had a variety of chat experiences online. &nbsp;Some of those experiences may have been chatting with human agents on the other end and some have been interacting with chatbots. Further still, some may have been a hybrid mishmash of automated and human responses. It is often difficult to glean from looking at a chat UI whether anyone&rsquo;s &ldquo;home&rdquo; on the other end, am I right?&nbsp;
Your top user experience priority number should be to set clear expectations about what topics your chatbot is trained to support. If your users have to discover the boundaries of productive conversation with your chatbot on their own, they&rsquo;ll get frustrated, give up, and maybe not come back. It should go without saying, but making it clear your chatbot is not a human agent is a helpful step so that your users don&rsquo;t have to figure that out through their own Turing tests.
As with any feature, adding in a new way of doing things on your site runs the risk of adding confusion for existing users who were used to doing it the old way and for new users who are now faced with multiple paths to take. &nbsp;They might wonder whether they should chat with the chatbot or use the existing UI to accomplish their goal.
The extent to which your chatbot will succeed will depend on your ability to comprehensively consider your users&rsquo; needs in context of the conversation they hope to have. &nbsp;How completely can your team train your chatbot on the subjects required in order to enable users to be successful? Do you have a means of supporting users who get stuck? &nbsp;How will you adapt to edge cases or unpredicted user scenarios?
<h2>The business impact of adding a chatbot</h2>
The presence of a chatbot in your application could positively or negatively impact your key metrics like retention rate and conversion rate. Or it could have no meaningful impact relative to the status quo.
Typical business goals we hear from Causal users related to chatbots are to:
<ul>
<li>Increase user retention</li>
<li>Increase conversion</li>
<li>Decrease support costs</li>
</ul>
<h3>Increase User Retention&nbsp;</h3>
Do users who see the chatbot / engage with the chatbot return more often than those who don&rsquo;t?
To compare the impact of a chatbot with the absence of one (the status quo) in an experiment, we&rsquo;d want to measure user retention and repeat usage rates like DAU / MAU for users. &nbsp;
In other words, when users see the chatbot and engage with it, does it significantly increase the likelihood that they&rsquo;ll be retained as users/customers?
<h3>Increase Conversion</h3>
Are users who see the chatbot / engage with the chatbot more likely to convert to a goal like an on-site task completion?
Chatbots can excel at collecting inputs from users in a natural language interface and presenting personalized responses that direct users towards a goal. &nbsp;This makes chatbots well suited to handle multi-step user flows such as an onboarding process. &nbsp;
Your app likely has one or multiple key goals that you&rsquo;re trying to achieve, such as completing a user profile, interacting with search results, or adding a credit card.
So a key metric you&rsquo;d want to track is the number of goal completion events that your chatbot users achieve relative to non-chatbot users.
<h3>Decrease Support Costs</h3>
Do users ask support questions that could be handled automatically?
Chatbots are great at responding to frequently asked questions, as long as the scope of topics is well defined. &nbsp;They&rsquo;re typically faster to respond and available 24/7.
On the other hand, there&rsquo;s a risk that your chatbot generates a response that is incorrect or inappropriate, leaving you with a larger support challenge.
<h2>Chatbot-specific analytics</h2>
To gauge the efficacy of your chatbot beyond high level business metrics, you&rsquo;ll likely want to capture metrics about the chatbot conversations itself, for example:
Conversation Depth: How many messages did the user send? &nbsp;Note that longer is not necessarily better, and you may have a target range depending on the user goal(s) that your chatbot supports.
Sentiment Analysis: How positive was the sentiment of the messages that the user sent? &nbsp;Nobody wants an angry chatbot user.
CSAT Score: What explicit satisfaction feedback are you able to capture in or after the conversation?&nbsp;
Binary outcome feedback request: You could also use a simple question &ldquo;Was this conversation helpful? [ Yes ] [ No ]&rdquo; rather than a scaled rating.
Link clicks: If your chatbot recommends links, what percentage of the time did users click on them? &nbsp;You need a way to determine whether those links were relevant, e.g. did the user stay on those links for a meaningful amount of time after they clicked? Or did they bounce and request a different result?
Errors: If your chatbot failed to respond with content or in a timely manner.
<h2>How Causal can help</h2>
Our customers use Causal to run LLM-powered chatbot experiments and measure outcomes in terms of the metrics that matter to them.&nbsp;
Wiring up the necessary events and doing the tracking to ensure your data is clean and usable for this kind of experiment is tedious work.
Causal can inject the user&rsquo;s context into your LLM prompts in a consistent manner so that it can be used to deliver more personalized and relevant responses.
If you&rsquo;re experimenting with a chatbot as part of your product and need a way to measure the success of your experiment, schedule a time to <a href="https://book.vimcal.com/p/christina/45minutes-0571c" rel="follow noopener" target="_blank">talk to us</a> or <a href="https://tools.causallabs.io/signup" rel="follow noopener" target="_blank">try our demo app</a>.

Overview We wrote this article for teams experimenting with chatbots as part of their product. We’re going to share potential goals, conside

Adding an LLM-powered chatbot to your application for fun and profit

Goals, considerations, challenges, and measurement tips to help you effectively test the impact that a new chatbot may have on your business and user experience.

<h2>Hi Paul!&nbsp;Could you please give me a bit of your career history?</h2>
I started my technical career doing full stack web development. I worked for a company called Airfare Watchdog that wanted to build a mobile app from scratch, so I learned iOS and now I've been doing iOS development for the last 10 years. I got into the native field by chance.
At Trip Advisor, I worked with (Causal Cofounder) Alex to maintain the TripAdvisor app. &nbsp;
I spent some time at Shopify working in react native and then I joined Causal.
Now I'm interested in getting back to more full-stack development. Working at Causal is a really good opportunity to do iOS, Android, and whatever else needs to be done!
<h2>What attracted you to working at Causal?</h2>
Causal is interesting because every company that I've worked for has built a version of a product to do what Causal does, either built something in-house or had something in-house and a third party tool. And a lot of them weren't really great, or they weren't strongly typed, or they were just poorly maintained, or they were built with a focus on the web.
It&rsquo;s exciting to offer a solution like Causal that works for all surfaces and would just easily work for pretty much any product development team.
An example of one of these homegrown A/B systems I worked with in the past just gave you a &ldquo;recipe&rdquo;, like do you want option A or B? &nbsp;It just returned a string. So, throughout the code, we were checking for this string everywhere. There was no way to know programmatically, what all the different variable options were. And so you're kind of guessing that you got the right recipe! Of course, the server could change out from under your code base and change what the recipes were. That's probably one of the scarier homegrown A/B testing solutions I&rsquo;ve seen, where the code you're writing, you don't know for sure if it's going to work.
In contrast to that, the Causal SDK for iOS right now is super strongly typed, which will help dev teams avoid some of the problems I&rsquo;ve had in the past.&nbsp;
Everything is super visible inside of your editor, for example auto-complete just populates with all the different options. You type client signal and you literally see all the things that you can signal on.&nbsp; It's just super slick now!&nbsp;
<img height="247" width="624" src="https://lh5.googleusercontent.com/eD50mcLCvDCR6U3QCl6dPw8rKYibFPZyBcmwbiz63S0QmhcVnuzNlWA0BH2awgLhmzdxiKoZ3Es9RwZdZWTIKdvVVIuon1YqmZjr2huQcgqWNglXqWZcrEpKbVm0eVPLetLv7XEXNsskiClwFJgC1zA">
So it's so easy for the developer to know exactly what the available tracking calls are for the given instance and then what parameters are required to make that tracking call.
And each parameter is again strongly typed and if it's required, you have to supply it. It keeps everything consistent across multiple calls. It makes it very easy for the developer to know if they're doing it wrong.&nbsp;
And if the signature changes to say, sometime down the stream. Say, somebody updates the event to have a new parameter that's required&hellip; The code won't compile and you have to make the changes to make the code work.
In contrast, if you were just tracking some JSON blob that wasn't strongly typed, you would never know that there's an issue with your tracking code at all.

<h2>What are some challenges you're most excited to solve for engineering teams using Causal?</h2>
While working on native mobile apps in the past, I've seen a ton of challenges with analytics tracking.&nbsp; In my past job, the web team had already built a huge tracking infrastructure geared for how websites worked.&nbsp; It was difficult to work with a native surface that has different navigation patterns.&nbsp; For example, when you navigate from screen A to B on the web, screen A is gone and it&rsquo;s a new screen view. In mobile apps, the paradigm is different.&nbsp; Users might go back and forth between screens and screens are persisted in memory, so the notion of a screen view is different.
Also while attempting to plug into that web-oriented framework, it was difficult to get all of the fields that were required to make the tracking work when we didn't have those fields in the native app. We had to do a lot of documentation on the data we were going to supply. Then it would be up to the downstream analysts and make sure that they knew the documentation. And when they were querying the data, that was terrible! They had to just know by the documentation that some things might be missing and not to rely on that if they're trying to compare native to web. It wasn't a good solution&hellip;
So, at Causal, we're working hard to keep our native SDKs feeling as native as possible. We want it to be easier for native developers to pick up, rather than something that feels web-focused. Causal's iOS SDK should feel like the other SDKs they are already familiar with. I would like to hear that they think that the SDK is clean and not &ldquo;ultra webby&rdquo;, that it is reliable, and that it&rsquo;s consistent with the native ecosystem.
<h2>Extra Credit Bonus Question... When you're not working on Causal, what do you do for fun?</h2>
When I'm not working, I enjoy spending time with my family, working on hardware and software side projects, experimenting with different recipes on my smoker, and hitting the local mountain bike trails.

Hi Paul! Could you please give me a bit of your career history? I started my technical career doing full stack web development. I worked for

Three questions with Paul, a Causal Engineer

Paul, an engineer at Causal, shares his thoughts on the platform and the Native iOS SDK he's building to help dev teams avoid the challenges of poorly-made, home-grown experimentation platforms.

<h3>Introduction</h3>
If you&rsquo;ve spent any time in the world of A/B testing, you've probably heard of terms like frequentist and Bayesian, fixed horizon and sequential testing, and the peeking problem. You might have even heard fairly technical definitions of the terms. In this article, we explain how these terms relate to each other. Here&rsquo;s the essential argument:
<ol>
<li>
A/B testing has traditionally relied on frequentist approaches
</li>
<li>
Frequentist approaches mean that you have a set sample size and can&rsquo;t peek at the results
</li>
<li>
There are a few alternatives to straight frequent &ldquo;fixed horizon testing approaches&rdquo;
</li>
<li>
One is sequential testing methods (still frequentist!)
</li>
<li>
Another is to go with Bayesian approaches
</li>
</ol>
Let&rsquo;s get started!
<h3 role="presentation">A/B testing has traditionally relied on frequentist testing</h3>
Traditionally, A/B testing has relied on frequentist rather than Bayesian testing methods, and if you&rsquo;ve ever used an A/B testing platform (including Causal), you&rsquo;re probably using a frequentist test.
This is the classic &ldquo;rejecting the null hypothesis&rdquo; paradigm where you use a T-test or Z-test to test a specific hypothesis and p-value against the null to determine statistical significance.&nbsp;
Frequentist methods assume the sample size is fixed before looking at the data. For example, the p-value from a T-test quantifies the probability of observing the results under the null hypothesis, assuming the sample size is fixed. If the sample size can change, then the significance levels lose their meaning.
<h4>Example</h4>
Experiment: You want to test a new email newsletter design against the existing design. The goal is to see if the new design increases the response rate. The existing design has a response rate of 3%.&nbsp;
<ul>
<li>
Null hypothesis: The new email response rate is equal to the old response rate of 3%.&nbsp;
</li>
<li>
Calculate the sample size that&rsquo;s necessary to achieve a certain power: To detect this amount of lift with 80% statistical power at the 5% significance level:
<ul>
<li>
Using a standard sample size calculator for proportions, with assumed response rates of 3% (p1) and 5% (p2), at 80% power and 5% alpha, the required sample size is 1187 recipients per variant.
</li>
</ul>
</li>
<li>
Send out ~2400 emails and measure the observed response rate
</li>
<li>
Perform a z-test to compare the new observed rate to 3%
</li>
<li>
If the p-value from the z-test is &lt; 0.05, reject the null and conclude the new rate is statistically significantly higher
</li>
</ul>
<h3 role="presentation">Frequentist, fixed-horizon testing means that you have a set sample size up front, and you can&rsquo;t peek&nbsp;</h3>
The peeking problem refers to repeatedly checking the results of a frequentist A/B test before it has reached the planned sample size. This violates the assumptions behind significance testing and leads to an inflated false positive rate.
When you check a test's results multiple times, you are essentially asking "Did the test ever show significance?", rather than "Is the test significant at this specific point?". Under the null hypothesis, the test statistic fluctuates randomly as more data comes in. By checking many times throughout the test, you increase the chance of catching one of these random fluctuations and incorrectly declaring significance.
For example, see the diagram below: the test statistic does a random walk where 5% of the time it reaches the significance threshold just by chance. If you check only once, the false positive rate is 5% as expected. But if you check 20 times, the chance it ever crossed the threshold due to random chance is much higher, around 40%. The more you peek, the higher the false positive rate becomes.
To avoid this problem:
<ul>
<li>
Determine the required sample size up front based on the desired statistical power
</li>
<li>
Wait until the test reaches the full sample size before checking for significance
</li>
<li>
Check for significance only once to control the false positive rate at the planned level (e.g. 5%)
</li>
</ul>
This practice of running an A/B test until a predetermined endpoint or "horizon", rather than continuously monitoring the test and stopping when statistical significance is reached, is called &ldquo;fixed-horizon testing&rdquo;.
<img height="301" width="624" src="https://lh6.googleusercontent.com/z1OdKacqPoNWN8hdAert7Ro6NZPMdX7OhGQ1Xk7E3SmMSCipyfocWRDanQFBLoOi6SUFooc7CPFx2kin1kXjcDXsaOHSlCITtgwocsznAtcIaD_89rWWoIxg8x3keg8209-HS17zzJnM2gY4iFgYew4">

<img height="405" width="624" src="https://lh4.googleusercontent.com/2Jk_WNd_6XczFWZyqLCQCZRtLpFPAjMREN8RGWVjY8UdqkOGVAprcEAi7XtRjEny1FULU7s7FeLINbc4Ph4mCySEucirdYdzZTTgYxm-CCDiMAX4sUTtut3QAs1RyhyFJ74RTuxzZGIYQDUTjRKhmIY">
<h3 role="presentation">An alternative to the traditional frequentist fixed horizon testing is Sequential testing</h3>
mSPRT, or Mixed Sequential Probability Ratio Test, is one alternative to the classic frequentist fixed horizon testing that Causal has implemented. It's a sequential analysis technique that allows for continuous monitoring of experiment results. Instead of waiting until the end of a predetermined sample size (as is typical in classical fixed-horizon hypothesis testing), the mSPRT allows for making decisions (e.g., stopping the test early) based on the data as it comes in. This can be particularly useful when looking to identify significant effects as quickly as possible and is especially relevant in online settings where user traffic and interactions are large and continuously flowing.
The key &ldquo;insight&rdquo; of sequential testing is that instead of performing a t-test against a fixed critical value at each analysis, it carefully controls how the Type I error allowance is spent over time.&nbsp;
Before the test begins, two boundaries are defined based on the desired Type-I error rate (often denoted by the alpha level, like 0.05). One boundary is for a statistically significant positive effect (upper), and the other is for no significant effect (lower). The boundaries gradually get narrower - in the beginning, the threshold for declaring significance is extremely stringent to avoid false positives. As more data comes in, the testing boundary loosens to enable stopping well-performing variations earlier.
<ul>
<li style="list-style-type: none;">
<img height="301" width="624" src="https://lh6.googleusercontent.com/3F9kZRuYTmvp4XkCQ0zFBFYpEMCK1eAXCPprS6vKRnuYsFR4PKYMkR5QiO7Lp5VY1WCy7XlYI17-hjhMU336hOMSrrVOrykqDJZ0frji-YlO23uPO4BgUnUxB22NxabFi3TLszSJnRs64nhj1gJ5J2Q">
</li>
</ul>
As the test progresses and data flows in, for each new data point, the log-likelihood ratio of the accumulated data is computed. This ratio gives a measure of how well the data fits one hypothesis relative to another.
At regular intervals or after each new data point, the cumulative log-likelihood ratio is compared to the predefined boundaries. If the ratio crosses the upper boundary, there's enough evidence to reject the null hypothesis, indicating that your treatment (variant B) is performing better than the control (variant A). If the ratio crosses the lower boundary, you conclude that there isn&rsquo;t a significant difference between the treatment and control, and you can stop the test early. If neither boundary is crossed, the test continues.
This flexible approach removes the need to pre-determine required sample sizes and experiment durations upfront. For modern tech companies that have data flowing into their databases in near real-time, it allows them to be able to analyze results not just at the end of a pre-defined experiment duration, but sequentially as the data accumulates day by day.
<h3 role="presentation" style="text-align: left;">The Bayesian testing approach</h3>
In more recent years, Bayesian A/B testing approaches have become popular, with A/B testing platforms like Optimizely implementing them.
Unlike frequentist testing, Bayesian methods assume a prior probability distribution based on existing beliefs and historical data.
<pre>P(B | A) = P (A| B) P(B)</pre>
<pre>posterior = likelihood x prior</pre>
The likelihood is continually updated as new data comes in, while the posterior provides probabilities of different outcomes. Since the likelihood/posterior are continually being updated, you don&rsquo;t have to set a sample size up front; instead, as you accumulate more data, the variance of the posterior distribution tends to decrease. This makes intuitive sense: more data makes the inference more precise.
Bayesian approaches are inherently sequential, and you can often get predictions with smaller sample sizes, but it&rsquo;s not true that you can just &ldquo;peek&rdquo; and stop the experiment whenever the results are what you want. Just as with the T-statistic, the Bayes factor does a random walk as data comes in, reinforced by the prior. Peeking biases the results by catching random fluctuations.
Furthermore, even if you run a Bayes A/B test with a fixed sample size, note that there are no guarantees about false positive rates.
Experiment: You want to test a new email newsletter design against the existing design. The goal is to see if the new design increases the response rate. The existing design has a response rate of 3%.&nbsp;
<ul>
<li>
Start with a prior belief of a 5% response rate for the new email (based on it looking nicer)
</li>
<li>
Model this as a beta distribution, e.g. Beta(15, 300) representing 15 responses out of 300 emails expected
</li>
<li>
As each batch of emails is sent, update the posterior distribution using Bayes' rule
</li>
<li>
Can calculate the probability that the new rate exceeds 3% after each update
</li>
<li>
May reach strong probability conclusions with fewer emails sent. For example, if after 100 emails, the posterior probability of the new rate exceeding 3% is already over 50%, then you can probably end the experiment.
</li>
<li>
Note that the result here is a probability rather than a definitive yes/no answer.
</li>
</ul>
<h3>Conclusion</h3>
Which approach and test should you actually use?
Many A/B testing experts and large tech companies like Uber have ended up using sequential, frequentist tests. This allows you to analyze results as they come in while still controlling type I error rates. However, Bayesian methods are gaining traction as well due to the intuitive probabilistic interpretation and ability to incorporate prior information.
Ultimately, the choice depends on your specific goals and constraints:
<ul>
<li>
If strictly controlling false positive rates is critical, sequential frequentist testing may be preferable. The overall alpha level is rigorously maintained
</li>
<li>
If you want probabilistic statements of evidence, Bayesian testing provides that with credible intervals and posterior probabilities
</li>
<li>
If you need to incorporate historical data or expert beliefs, a Bayesian approach leverages that well through the prior
</li>
<li>
If sample sizes are limited, Bayesian tests may provide more power, but results should be interpreted with care
</li>
<li>
For users without statistical expertise, frequentist methods and their classic hypothesis testing framework may be easier to understand
</li>
</ul>
There is no universally superior approach; each has pros and cons. Regardless of the method you use, peeking at results early and stopping at random times invalidates the guarantees. Proper experimental design and discipline against peeking is critical for reliable A/B testing.

Introduction If you’ve spent any time in the world of A/B testing, you've probably heard of terms like frequentist and Bayesian, fixed horiz

Which statistical approach should you use for A/B testing?

Frequentist, Sequential, and Bayesian testing approaches

<h2>Introduction</h2>
As companies integrate LLMs into their existing products, they&rsquo;re writing increasingly sophisticated and dynamic prompts. With &ldquo;context stuffing&rdquo;, they might provide real-time product context as input to the LLM, and then take the LLM output and re-incorporate that into the product UI or experience.
Let's take the simplest example - that of a customer service chatbot. When a customer initiates a chat conversation with a company's customer service bot, the chat interface sends the customer's initial question to the LLM API along with key metadata like:
<ul>
<li>
Customer name, account info
</li>
<li>
Previous chat history/transcript
</li>
<li>
Page they were on when they started chat
</li>
<li>
Any order details if on a checkout/account page
</li>
</ul>
The LLM processes this context and generates a response attempting to answer the customer's question or resolve their issue. The response text is then returned to the chat interface and displayed to the customer as a bot reply.
Alternatively, take an e-commerce example: when a customer is browsing a product page on an e-commerce website, the key details of that page are fed into the LLM API, including:
<ul>
<li>
Product title, description, price, images
</li>
<li>
Inventory status
</li>
<li>
Customer browsing history and purchases
</li>
<li>
Related/recommended products
</li>
</ul>
The LLM uses this context to generate relevant text for the page, including:
<ul>
<li>
An overview paragraph highlighting the product's key features and benefits
</li>
<li>
Bullet points summarizing specs and details
</li>
<li>
Responsive sizing recommendations based on customer data
</li>
<li>
Answers to frequently asked questions
</li>
<li>
Reviews or ratings selected from the product's existing data
</li>
<li>
Complementary products the customer may also like
</li>
</ul>
All this text is returned to the product page and displayed in the appropriate locations - overview section, details tab, sidebar recommendations, etc. The responses fit seamlessly into the existing page layout.
When the user experience relies on inputs to the LLM, new kinds of challenges arise: How should you refine and test different prompts to get better results? How do you measure the real-world impact of different prompt or different models on your customer experience?
In this blog post, we first discuss some of the top resources for prompt engineering and then explore Causal&rsquo;s support for LLM experimentation to make it easy to A/B test prompts and write more interesting, complex prompts with user context.
<h2>Tools for Experimenting with LLMs and Prompt Engineering</h2>
<h3 role="presentation"><a href="https://openai.com/blog/function-calling-and-other-api-updates">OpenAI Function Calls</a></h3>
Function calling is a new OpenAI feature. It enables the models to parse out structured data from unstructured natural language, and pass this structured data as parameters to functions that you have defined. This is crucial if you want to retrieve relevant data from data sources or API&rsquo;s or integrate with other systems or tools.
Here&rsquo;s an example:
Let&rsquo;s say you want to answer something like, &ldquo;What&rsquo;s the weather in San Francisco right now?.&rdquo;
With function calling, the model will parse out that the location desired is San Francisco, CA. It will then pass it along to a get_weather_at_location function that you have defined, and return an accurate weather forecast (rather than one the model hallucinated).
Function calls can also translate natural language to internal SQL queries and database calls. As you look to fetch additional context from other data sources and add it to your LLM inputs, function calling is an important concept to master.
<h3 role="presentation"><a href="https://github.com/hwchase17/langchain">LangChain</a></h3>
Langchain is an open-source framework that simplifies the creation of applications using LLMs. It provides a standard interface for interacting with LLMs and several tools for building applications, including chatbots, question-answering systems, and creative text-generation tools.
The library's core concept revolves around the ability to "chain" various components to create more sophisticated applications using LLMs.&nbsp;
Langchain offers a variety of functionality for working with prompts, including prompt templates and tools like output parsers and example selectors. It enables structured interactions with LLMs.
<pre>const promptTemplate = "Translate the following phrase: '{phrase}'";</pre>
You can use this template to generate prompts like:
<ul>
<li>
Translate the following phrase: 'Hello, how are you?'
</li>
<li>
Translate the following phrase: 'Where is the nearest coffee shop?'
</li>
</ul>
<h2><a href="https://www.vellum.ai/">Vellum</a></h2>
Vellum is a developer platform for building on top of LLMs. It aims to bring MLOps-type process and automation to LLM experimentation and deployment.
Vellum consists of four components - Playground, Search, Manage, and Optimize. The components that will perhaps be most useful to you are Playground, Search, and Manage.
Playground: a UI that enables you to perform side-by-side comparisons of prompts, parameters, models, and even model providers across a variety of test cases.
<img height="164" width="624" src="https://lh3.googleusercontent.com/HfeTPIqZkF3212mXsRnKK8IPUefyeCoL7RdIAxKmuzUR5UIKU2tHEfCJzEn2ywoXCJKwlqGy_Jtt-DPFPMpgonPkcNsBr1l2bh97eUN3rSNgmofC2Nx2G2IdZe9Z6_IbiHPct_r0KoMIKiYv9AjEaPA">
The screenshot above uses the Vellum Playground to compare two prompts and their respective models for a text summarization application.
Search: while we have primarily discussed deterministic ways of adding context to prompts - i.e. fetching additional user data from a table, an alternative approach for more &ldquo;fuzzy&rdquo; context, i.e. parts of a documentation site most likely to be affected by a code change - is to upload an entire corpus of text (e.g. your company documentation), and convert the text to embeddings. Then, when the LLM call is made, the most relevant embeddings are injected as context into the query. Vellum makes this easy to do with its Search component.
Manage: Vellum also offers an API wrapper around the OpenAI, Cohere, and Anthropic models that provides full observability into what you are sending and receiving from these models. You can replay historical requests and see version history.
<h3 role="presentation"><a href="https://github.com/jordanful/promptspot">Promptspot</a></h3>
Promptspot enables you to test prompts against numerous inputs (your application data) in a single test suite and to collaborate on prompts with your team via version control and drafts.
The screenshot below, for example, demonstrates a &ldquo;Managing Travel Expense&rdquo; prompt tested against a &ldquo;Booking to San Francisco&rdquo; input using the text-davinci-003 model.
<img height="320" width="624" src="https://lh3.googleusercontent.com/E5oIyg23bRWCSB3hGGUwDPj9LossdsUg7YruHR7JdGrLFltnkREH2hDgiUXNgpF57M4F77tQgV76gK4i3WKl1hGWEIr_VRmdyXqC_M0r0faFcGJWVYlf4rCL7KAFcBIGIs_ElWm291MUj7KKF6_7TdI"> 
<h3 role="presentation"><a href="https://ozoneml.com/">Ozone</a></h3>
Ozone simplifies the process of prompt creation, unit testing and validating AI interactions.
The following test in Ozone ensures that the name and role of a colleague are mentioned while introducing them in an email. You can also try the same prompt with different models by modifying the settings in Ozone.
<img height="296" width="561" src="https://lh5.googleusercontent.com/uYwAUp-j00ty74zm_E1ZGJwd-XIPhfXbWcO3FHUpZ_l6UjMvgFJ5hKYUlw23b7vWAu8eZko5bfCwZdw5RqLOeIqlpabCo9US2PmeV8RMCodcwJTOchvO06RzpwJiN28mz0OughUyxS7_0XkqEIlmk8Q">
<h3 role="presentation"><a href="https://flux.paradigm.xyz/">Flux</a></h3>
Flux employs a tree structure to generate multiple completions per prompt, enabling users to explore various creative responses. With Flux, users can test different prompts while maintaining a shared context, facilitating experimentation and finding the most effective inputs for the LLM.&nbsp;
The screenshot below is an example of multiple completions for the same prompt.
<img height="329" width="624" src="https://cdn.buttercms.com/JUNRPUIS7iwY2btY4YTA">
<h3 role="presentation"><a href="https://humanloop.com/">Humanloop</a></h3>
Generative models are much more subjective than traditional machine learning tasks like classification, where either the model predicted the correct class or it did not. With use cases like copy writing or customer support, it&rsquo;s less clear how well your model is doing. One potential solution is just to ask your customers - which LLM output do they like better?
Humanloop makes this possible by offering an SDK that is a drop-in replacement for OpenAI&rsquo;s API&rsquo;s. This SDK enables you to not only call OpenAI&rsquo;s API&rsquo;s, but also logs every call and <a href="https://docs.humanloop.com/docs/capture-user-feedback-using-the-sdk">captures explicit user feedback to help evaluate whether the LLM outputs were good</a>. This makes it possible to go directly to your users and optimize for user outcomes and preferences.
<h2>LLM experimentation with Causal</h2>
While Humanloop takes a survey-like approach to model evaluation, the reality is that most customers probably won&rsquo;t want to answer pop-up questions while browsing a site. Companies with large user bases will want just to put the results of different prompts in front of their customers and determine which ones perform better implicitly via A/B testing and experimentation. This is where Causal can help.
Causal is an end-to-end feature flagging and experimentation platform. Recently, we&rsquo;ve added experimentation infrastructure that makes integrating LLMs into your product and optimizing their performance easy. Here's how it works:
<h3>Swap in LLMs with Feature Flags</h3>
Causal enables you to toggle different LLMs or versions of models on and off for different users. Rather than needing engineering work to swap models, you can use Causal's visual editor to point your product to different LLM endpoints. This makes it easy to A/B test models like GPT-3.5 vs Anthropic's Claude model.
<img src="https://cdn.buttercms.com/C5HdHjJRVidoQVBFlfAh" alt="undefined" width="666" height="504">
<h3>Experiment with Prompts</h3>
Fine-tuning prompts is critical for getting good results from LLMs. With Causal, you can test out different prompts and parameterizations and see which ones perform best. Causal will serve different prompt variations to different users and report back on how they impact metrics like engagement, conversation length, etc.
<h3>Add User and Context-Awareness</h3>
Basic prompts often lead to generic, repetitive responses from LLMs. Causal enables you to include user attributes and contextual data to make prompts more specific. For example, you could customize an e-commerce chat bot prompt with the user's name, past purchases, and current page they are on.
Causal takes care of routing all this context data to the LLM and returning responses back to your product. This makes it far easier to build contextual, personalized LLM experiences.
<img src="https://cdn.buttercms.com/cXRuTFIvQPClUpMT9HpZ" alt="undefined" width="666" height="243" style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;">
<h3>Measure Real User Impact</h3>
With Causal's built-in experiment reporting, you get clear insights into how adjustments to models, prompts, and parameters affect key metrics. This enables teams to rapidly improve their LLM integrations through rigorous experimentation.
<h3>Conclusion</h3>
For companies that are trying to integrate LLMs into their products, a common workflow is to feed inputs into a machine learning model, get a response, and feed the response back into the product. Causal dramatically simplifies the implementation of this workflow, giving teams the ability to successfully experiment with, optimize, and scale LLMs in their products.

How Causal (and other tools) can help your team experiment with LLMs and prompt engineering Introduction As companies integrate LLMs into th

Tools to help your team experiment with LLMs and prompt engineering

Resources for teams experimenting with LLMs

<h2>Common issues</h2>
Gathering complex training data
Traditional product data systems are designed to gather point-in-time data (e.g. a user click, a pageview, etc), but the data needed to train many recommender systems is more complex. Typically, what&rsquo;s needed is the model results, the state of the user (and any other model inputs) at the time those results are shown, and any downstream payoff events (user clicks, purchase, etc). Tying all that data together requires complicated, time consuming ETL work. Those ETLs are fragile and require updates as the product shifts. Plus, it&rsquo;s very easy to introduce time shifting errors into training data when writing ETLs manually, which allows models to peek into the future and ruins their performance in production.
Integrating model results back into the product and evaluating their effectiveness
While there are many options for deploying ML models into the cloud and making them available as an API endpoint (e.g. Sagemaker, Seldon), taking those models the last mile and incorporating their predictions into the product (i.e. displaying them to users) typically requires front-end engineering work. As a result, many ML teams spend too much time waiting to be able to evaluate whether their model delivers real results for users and the business. Traditional feature management and A/B testing systems aren&rsquo;t capable of handing A/B testing for ML models out of the box.&nbsp; This slows down machine learning teams and adds unnecessary overhead.
<h2>How Causal helps</h2>
Causal helps machine learning teams learn quickly and focus their efforts on the interesting part of the work: building and iterating on the models.&nbsp; Specifically, Causal:&nbsp;
<ul>
<li>
Makes it easy to create and reuse model features&nbsp;
</li>
<li>
Allows the ML team to ship new models to customers without support from the rest of the engineering team&nbsp;
</li>
<li>
Serves the model inputs needed at run time, and returns predictions to the product with very low latency
</li>
<li>
Offers out of the box A/B testing for ML models, using any kind of business metric needed
</li>
<li>
Logs all the data needed for future training runs, in exactly the format needed
</li>
<li>
Improves collaboration between ML team members and engineers by providing a tool to manage the intersection between an application&rsquo;s user-facing front-end and the ML models that power them
</li>
</ul>
<h2>How it works</h2>
First, you define the data model needed to train, run, and evaluate your model.&nbsp; The data model includes three types of data:&nbsp;
<ol>
<li>
Inputs, which are the features in your model. These could include things like a snapshot of the user&rsquo;s most recent views or purchases, the ID of the item the user is looking at now, or recent searches.&nbsp; Causal offers a plugin architecture that allows you to easily call out to any data system you have to gather model inputs.&nbsp; Plugins are a way to access data from any data system at run-time (either to display to users or feed into a model).&nbsp; Plugins are arbitrary code that allow you to call out to external systems or just apply logic to fill in the inputs to your ML model. Once an input is defined, it can be reused as a feature in any future model.
</li>
<li>
Outputs, which is just the structure of your model&rsquo;s outputs, e.g. an array of product IDs returned by a recommender or the text returned by an LLM.&nbsp; The data model is written using GraphQL&rsquo;s data types, so Causal can support virtually any sort of output.
</li>
<li>
Events, or your payoff. These are used to record how users respond to your model predictions (e.g. do they click on recommended items) and are used to create metrics to evaluate your model&rsquo;s effectiveness (e.g. revenue per user).
</li>
</ol>
Second, you integrate Causal into your product&rsquo;s codebase, calling out to get needed outputs and to record events. You only need to do this once for a given part of your product. Once it&rsquo;s done, you can test and iterate indefinitely without needing to update the codebase again.
From there, Causal enables you to link any new model to your product without changes to your codebase, meaning you can swap new models into your product, ramp up traffic to them, and A/B test them independently. Then, at run time, Causal will:&nbsp;
<ul>
<li>
Gather all your inputs and feed them to your model
</li>
</ul>
<ul>
<li>
Return the model predictions to your product
</li>
<li>
Record all the data you need in the perfect format: one row per impression of your model, including all your features, your predictions, and your payoff
</li>
<li>
Evaluate your model's performance against performance metrics (e.g. NDGC, recall, MRR) and business metrics (e.g. click through rate, revenue per user)
</li>
</ul>
As you build winning models, you can roll them out to your users through Causal, without needing to update your codebase.

<h2>Example</h2>
As an example, imagine you want to recommend job listings to a user based on:&nbsp;
<ul>
<li>
The job the user is currently viewing
</li>
<li>
The jobs the user previously browsed
</li>
<li>
The user&rsquo;s resume
</li>
<li>
The user&rsquo;s location&nbsp;
</li>
</ul>
<img height="212" width="624" src="https://lh4.googleusercontent.com/PtvM3h2pPjiakbexMvPjUSOxISWhDz9674THTFscSfrH74D5IdxP5ztD30fvoW-zwU7fJy_H8-mWF5yR8TGWcueaee9P0fgW2G_VLuEhQZQwohQ7My6JKLkBsWedrgQUpiA2QmyAKQjsN2F6tmmO8Gs">
The features (inputs) of this model will be:&nbsp;
<ul>
<li>
The embedding of the current job
</li>
<li>
The embeddings of their browning history
</li>
<li>
The embedding of the user&rsquo;s resume
</li>
<li>
The user&rsquo;s lat/long
</li>
</ul>
The model recommendations (outputs) are jobs to show to the user, and the model&rsquo;s payoff is clicks.

Inputs
Each of the model&rsquo;s main features is setup as an input in Causal:&nbsp;
<ul>
<li>
The job they&rsquo;re currently viewing is passed in from the front end
</li>
<li>
A plugin gets the embedding for the current job and the user&rsquo;s browsing history
</li>
<li>
Another plugin gets the user&rsquo;s resume embedding and location
</li>
</ul>
The benefit of using plugins to gather the input data for a model is that Causal logs their values at run-time. This means you don&rsquo;t have to reconstruct that data later on (which is tedious) or use current snapshots (which allows models to peek into the future).
<pre>feature Recommendation { &nbsp;args { &nbsp;&nbsp;&nbsp;sourceItem : ID &nbsp;&nbsp;} &nbsp;plugin java "ItemEmbeddings" { &nbsp;&nbsp;&nbsp;sourceItemEmbedding : [Float!]! &nbsp;&nbsp;&nbsp;browsingHistory : [ID!]! &nbsp;&nbsp;&nbsp;browsingEmbedding : [Float!]! &nbsp;} &nbsp;plugin java "UserInfo" { &nbsp;&nbsp;&nbsp;resumeEmbedding : [Float!]! &nbsp;&nbsp;&nbsp;"lat / long" &nbsp;&nbsp;&nbsp;location_lat : Float! &nbsp;&nbsp;&nbsp;location_long: Float! &nbsp;&nbsp;}</pre>

Outputs
The output for this model is simply a list of jobs to recommend to users.&nbsp;
<pre> output { &nbsp;&nbsp;&nbsp;"List of jobs shown the the user" &nbsp;&nbsp;&nbsp;recommendations : [Document!]! &nbsp;}</pre>
The recommendations here are returned as document objects with calculated features that can be used to tune your ranking model. For example, you can use the similarity metrics in a subsequent learn-to-rank model to determine the optimal mix for those features.&nbsp;
Since you wouldn&rsquo;t want to send those metrics to the browser, you can add the @hidden directive, which will keep that information inside your data warehouse. The document&rsquo;s description is a long chunk of text that doesn't need to be stored multiple times in the warehouse, so using the @no_log directive allows you to pass the information onto the front end for rendering without recording it in the data warehouse.
<pre>type Document { &nbsp;item : ID! &nbsp;description : String! @no_log &nbsp;browsingSimilarity : Float! @hidden &nbsp;resumeSimilarity : Float! @hidden }</pre>

Events
The payoff for this model is user clicks on the job recommendations. Causal will tie these together with all your model inputs and outputs automatically.
<pre> event Click { &nbsp;&nbsp;&nbsp;itemId : ID! &nbsp;} }</pre>

Gathering training data&nbsp;
Prior to training your first model using these new inputs, the outputs can be set through Causal&rsquo;s UI (either a static list or a list sourced from any API endpoint) or you can continue to set them as you currently do.
At run-time, Causal will gather and log all your inputs and all your events. These are logged in exactly the format you need to train your model: one row per impression, with the inputs, outputs, and payoff all together.&nbsp;
Experimenting with a new model
When you&rsquo;ve trained your new model and are ready to test it in production, it&rsquo;s easy to roll it out to users through Causal. Causal integrates with model hosting tools like Sagemaker, and new models can be integrated into your product without needing to rely on front-end engineers or deploy new code. Once you have your model in an endpoint, Causal will fetch outputs from it.

<img height="346" width="449" src="https://lh5.googleusercontent.com/IAekz0umtgW59rTMHA22Huqg1bCnx1F8c9jXuvx_53Ky8ms4M059rvnb6_OQ3fB_fy1WHNyGqDOGIrhbdqCLzbTK2LoMgVarKMfvht01iQq3ho4AXMf8-dTRghTC04iBePXFl26zEIXmpWHdKofxfWM">
At run-time
Once you launch your new model to users, Causal will handle all the data brokering needed in production:&nbsp;
<ul>
<li>
Collecting your model inputs&nbsp;
</li>
<li>
Serving those inputs to your hosted model&nbsp;
</li>
<li>
Returning your model&rsquo;s outputs to your product&nbsp;
</li>
</ul>
Causal can either run on prem or in the same availability zone as your product, so all this brokering happens with very low latency.
Logging
As Causal passes all the data back and forth, it continues to log it all into your own data warehouse.&nbsp; Future training runs are very straightforward, with no ETL work required. In addition, if you choose to launch your models as an A/B test, Causal will monitor the performance of your models in production and allow you to roll out winning models easily.

<img height="387" width="624" src="https://lh4.googleusercontent.com/ZrL-b0tBPp6qIer1qwNYThRUry0fpMZjSl9E4tYoCE7VUt3GAujWuHThtJ0kgqS1MwTcvgub_dGHJ5TYCwcvfSgE8s-CMdkzfb9rStMzJq88zGSHUthLFKHn6PptawoUbquNEz9MRDRRLpivQbL0JC8">
<h2>Conclusion</h2>
Causal is designed for teams that use machin e learning models to improve their products. It handles the most tedious and time-consuming parts of ML work: recording and assembling training data, incorporating models into the product itself, and collecting and serving data to models at run-time in production. Using Causal allows data science teams to focus their time on the highest value work: developing and refining ML models that improve their product and grow their business. To experience the benefits of Causal firsthand, consider trying out Hello, Causal or setting up a demo today.

<h2>Appendix: Entire FDL File</h2>
<pre>"Recommended jobs for a given user on a given page" feature Recommendation { &nbsp;args { &nbsp;&nbsp;&nbsp;"The ID of the job the user is currently viewing" &nbsp;&nbsp;&nbsp;sourceItem : ID &nbsp;&nbsp;} &nbsp;"Embeddings for user's browsing history" &nbsp;plugin java "ItemEmbeddings" { &nbsp;&nbsp;&nbsp;sourceItemEmbedding : [Float!]! &nbsp;&nbsp;&nbsp;browsingHistory : [ID!]! &nbsp;&nbsp;&nbsp;browsingEmbedding : [Float!]! &nbsp;&nbsp;} &nbsp;"User info, including the embedding for their resume" &nbsp;plugin java "UserInfo" { &nbsp;&nbsp;&nbsp;resumeEmbedding : [Float!]! &nbsp;&nbsp;&nbsp;"lat / long" &nbsp;&nbsp;&nbsp;location_lat : Float! &nbsp;&nbsp;&nbsp;location_long: Float! &nbsp;} &nbsp;output { &nbsp;&nbsp;&nbsp;"List of jobs shown the the user" &nbsp;&nbsp;&nbsp;recommendations : [Document!]! &nbsp;} &nbsp;"User clicks on a recommended job" &nbsp;event Click { &nbsp;&nbsp;&nbsp;itemId : ID! &nbsp;} }</pre>

Optimizing recommender systems with Causal Common issues Gathering complex training data Traditional product data systems are designed to ga

Common challenges for ML teams and how Causal helps

Optimizing recommender systems with Causal

<h2>Overview</h2>
This article covers three different types of capabilities that product development teams need:&nbsp;Feature Flagging, A/B Testing, and Session/Event Tracking. We&rsquo;ll detail out some popular options for each category and explain how Causal supports each one.&nbsp;
This article is intended for product and engineering leaders who are evaluating tools in any of these categories and who are looking for ways to improve the pace at which their teams run experiments.

<h2>Feature Flagging Overview</h2>
Feature flagging enables product development teams to manage feature releases so that they can make new or experimental features available to different audiences.
There are numerous benefits of feature flagging:
<ol>
<li>
Release code safely: Separating feature visibility from the code deployment process. It&rsquo;s helpful to be able to put code into production and then make it available at a different time, in a controlled manner.
</li>
<li>
Run Experiments: Making different versions of an application available to different target user audiences, in order to understand how the feature impacts key business metrics
</li>
<li>
Build more quickly: Feature flags enable your team to parallelize the work of building the feature (coding) versus measuring the impact of the feature (analyzing) by giving non-technical team members better experimentation and analytics tools.
</li>
</ol>
Adding feature flagging does not come without challenges. It does increase the complexity of your code, and can make testing and QA more difficult.&nbsp; If feature flags are not implemented carefully, they have the potential to cause poor user experience (e.g. slow page loads) or even to expose sensitive data to the wrong audiences. The benefits of adding feature flagging capabilities outweigh the drawbacks for larger product development teams working on applications with a high volume of users.

<h2>Feature Flagging Software Examples</h2>
There are many feature flag software options on the market, each with their strengths and weaknesses. Here are a few examples to consider:
LaunchDarkly offers targeting, segmentation, and rollouts for feature flags, offers a range of SDKs for different programming languages, and provides integrations with a multitude of development and deployment tools.
ConfigCat is a feature flag and configuration management tool that enables teams to release features gradually and manage configurations across different environments.
LaunchControl is a feature flagging and remote configuration platform that allows teams to control the release of features and configuration changes. It offers targeting options and analytics to measure the effectiveness of feature flags.
Split.io provides feature flagging and feature experimentation capabilities. Split enables businesses to test different variations of their features on different groups of users.&nbsp;

<h2>How Causal Compares To Other Feature Flagging Platforms</h2>
One issue with a feature flagging system is that flags tend to accumulate over time.
Causal&rsquo;s approach to feature management is slightly different.&nbsp; Causal developed a feature definition language (FDL) that enables you to specify your features in code so feature changes can be tracked via common source control methods like Git. We found that, while it requires some up-front thinking to define and encode your features, our customers are extremely happy with the resulting downstream data cleanliness.
Most feature flagging software is centered around "on/off switches" for specific features. They effectively serve as if/else gates in your code. In contrast, Causal enhances your features by making them configurable. Configurable features are more stable than simple feature flags because non-technical team members can do many experiments without adding more junk into the code. This also means that, if your non-technical team members want to change the configuration of a feature (e.g. button copy, number of search results, etc.), they can do so without dev support.&nbsp; With most feature flagging software, your engineering team would need to create a new flag, write some code, go through the release process, and then enable the flag.
The limited nature of most feature flagging software often leads to nested spaghetti code with branching logic. The codebase becomes harder to understand.&nbsp; Feature flags linger in the code base because no one remembers to clean them up. In contrast, Causal puts deprecation warnings straight in the IDE to alert developers that it&rsquo;s safe to clean up unused experiments in the codebase.
<h2></h2>
<h2>A/B Testing Software Overview</h2>
A/B testing is a way to compare different user experiences. Typically, the goal of A/B testing is to optimize an application toward a key metric, like user engagement or site conversion. By creating and comparing multiple versions of an application, product dev teams can learn quantitatively what version works best to achieve their business goals.
Effective A/B testing requires that teams have a clearly defined testing approach and measure of success. A/B tests can go wrong in a lot of ways, for example:&nbsp;
<ul>
<li>
the team isn&rsquo;t able to agree on the definition of success or isn&rsquo;t committed to investing the effort required to properly test
</li>
<li>
the code isn&rsquo;t properly instrumented relative to key metrics
</li>
<li>
the testing approach is flawed, e.g. users aren&rsquo;t assigned in a randomized manner or there&rsquo;s bias in the way audiences are assigned to different test versions
</li>
<li>
the measurement approach is flawed, i.e. statistics aren&rsquo;t well understood or applied correctly
</li>
<li>
the feature in question doesn&rsquo;t have enough usage to collect significant data in a timely manner
</li>
</ul>

<h2>A/B Testing Software Examples</h2>
Here are some examples of software platforms that product dev teams can use to run A/B tests.
Split.io: Split.io is a feature delivery and experimentation platform that enables teams to manage feature flags and perform A/B tests. It provides real-time analytics, custom targeting rules, and rollbacks for controlling feature releases. Split.io supports multiple languages and frameworks.
ConfigCat: ConfigCat is a feature flagging and configuration management tool that helps teams control and manage their application features. It offers a simple API for implementing feature flags and supports various targeting options, including user segmentation and gradual rollouts.
Optimizely: Optimizely is a well-known A/B testing and experimentation platform. It provides a user-friendly interface for creating and managing A/B tests, allowing you to test different variations of your website or application. Optimizely offers features like visual editing, audience targeting, advanced analytics, and personalization capabilities.
Google Optimize: Google Optimize is a free A/B testing and personalization tool offered by Google. It integrates with Google Analytics and allows you to create A/B tests directly within the platform. Google Optimize provides a visual editor, targeting options, and basic reporting to help you optimize your website based on user behavior and data.
VWO (Visual Website Optimizer): VWO is a comprehensive A/B testing and conversion optimization platform. It offers a visual editor, advanced targeting options, heatmaps, session recordings, and goal tracking to conduct experiments and improve user experience. VWO also supports multivariate testing and personalization.
AB Tasty: AB Tasty is an all-in-one experimentation and personalization platform. It provides a visual editor, targeting options, segmentation capabilities, and advanced analytics to optimize your website or application. AB Tasty also offers features like funnel analysis, session replay, and customer journey analysis.
Adobe Target: Adobe Target is part of the Adobe Experience Cloud suite and provides A/B testing, personalization, and automated optimization capabilities. It offers advanced targeting options, AI-powered recommendations, and integration with other Adobe products for a comprehensive optimization workflow.

<h2>How Causal Compares To Other A/B Testing Platforms</h2>
Causal&rsquo;s integrated data model means your data will never look different between the front end and your data warehouse.
Because your feature specifications are defined in your code repository, your whole team has a shared, source-controlled document to refer to when discussing your experimentation approach.
Once features are defined using our Feature Definition Language, non-technical team members are able to experiment on any combination of feature attributes without further intervention from engineering.
The Causal feature schema also enables a much better experience for developers, because our SDKs are type safe. This enables easy code completion in an IDE and also provides useful compile-time errors to prevent downstream issues.

<h2>Session Management &amp; Event Tracking Overview</h2>
Product dev teams use session tracking software and event tracking software to analyze the way users interact with their applications. By analyzing patterns of use, teams can optimize the user experience, focus product development efforts, and more efficiently achieve business outcomes.
Some of the benefits of using session management and event tracking software are:
<ul>
<li>
Gathering usage data to inform decisions about product development
</li>
<li>
Identifying application performance issues that negatively impact the user experience&nbsp;
</li>
<li>
Finding bottlenecks in user flows that negatively impact business performance, e.g. barriers to conversion events
</li>
<li>
Train ML models
</li>
</ul>

<h2>Session &amp; Event Tracking Software Examples</h2>
Segment is a customer data platform that helps you collect, manage, and route data from various sources, including event tracking in web applications. It acts as a data integration hub and allows you to send event data to multiple analytics tools, including Google Analytics, Mixpanel, Amplitude, and others.
Heap Analytics is an event-based analytics tool that automatically captures every user interaction on your web application. It provides a visual analytics interface to explore user behavior, perform retroactive analysis, and create funnels. Heap Analytics offers features like event visualizations, cohort analysis, and user segmentation.
Snowplow Analytics is an open-source event tracking and data collection platform. It allows you to track and collect event data from your web application, which you can then analyze and process using your preferred analytics tools.

<h2>How Causal Compares</h2>
Because of its usage of Feature Definition Language, Causal prevents event definitions from spinning out of control. Your team has one source-controlled schema that defines your features. This feature definition schema ensures that your whole team &ndash; across engineering, product, and data science &ndash; is on the same page about what events are being tracked, what features matter, and how results should be analyzed.&nbsp;
Since data definition and governance is baked into your development processes, code can&rsquo;t go to production without the team considering the data governance implications of changing a feature. Other tactics like wiki pages and spreadsheets quickly become out of sync and require that team members spend significant time keeping documentation up to date.
Causal&rsquo;s automated ETL makes analysis of session, event, experiment, and feature data simpler, right in your own data warehouse. The hardest part of analyzing an experiment is making sure the data accounting is correct. Did this user really see a given feature? If an event was fired, was it associated with the first time the user saw the feature or the fifth time? Causal does all the hard work of keeping track of events and feature impressions so your team can focus on understanding the impact on your users, not data pipeline grunt work.&nbsp;
Causal goes far beyond other feature flagging systems. Other systems let you turn things on or off. Causal gives you fine grained control over how your features behave in production. With Causal your team can make configuration changes without code changes, test them and roll them out with zero downtime and no code deployment. When coupled with our robust data tracking capabilities, we&rsquo;re perfect for data science teams that need to AB test their models and understand the real user impact of their changes.
Read more in our <a href="https://tech.causallabs.io/docs/data-warehouse/overview" rel="follow">tech docs</a>.<a href="https://tech.causallabs.io/docs/data-warehouse/overview"></a>
<h2></h2>
<h2>Summary&nbsp;</h2>
There are a lot of choices out there when it comes to feature flagging, A/B testing, and session/event analytics. We built Causal to help you avoid stitching together half a dozen different platforms to achieve your product development team&rsquo;s experimentation goals.
Causal helps your team move faster and make better decisions over longer time horizons. Teams make better decisions when 1) they trust their data and 2) they have the right statistical tools to evaluate their data. Other tools might feel fast at first, but quickly bog the team down in clean-up and data cleansing tasks that ultimately distract from their core mission: make the product better for your users. Causal helps ambitious teams hit their goals without trading off speed for safety.&nbsp;

Overview This article overviews three different types of capabilities that product development teams need: Feature Flagging, A/B Testing, an

The landscape of experimentation analytics

Feature Flagging, A/B Testing, and Session/Event Tracking

The landscape of experimentation tools

<h3>Introduction</h3>
Google <a href="https://support.google.com/optimize/answer/12979939?hl=en">recently announced</a> they are sunsetting Google Optimize 360, their no-code A/B testing tool. While Google Optimize was popular and Google should get credit as the one major tech player to offer an A/B testing platform with a free plan, there&rsquo;s one aspect of Google Optimize&rsquo;s design that we won&rsquo;t be mourning.
That&rsquo;s the visual editor.
In the context of experimentation or A/B test tools, visual editors are user-friendly interfaces that allow users to make changes to a webpage without needing to write or manipulate code. Users can modify elements such as text, images, buttons, and layouts directly, and see the results of those changes in real-time. This is particularly useful for non-technical users, such as marketers or business owners, who want to make simple changes to a webpage for the purpose of A/B testing.
For instance, you might change the color of this &ldquo;GET AWAY FROM IT ALL&rdquo; header from red to green or reword it. The visual editor would allow you to make these changes simply by clicking and editing, rather than having to dig into the HTML or CSS code.
<img height="316" width="624" src="https://lh3.googleusercontent.com/hFVWRNsLKA06MgboVcHyA5KuEZ8dJ_wSQDSdBwTSRbWTxC8on14jDtXHwO-aLKo8-UhcBug0qR7IErPam61_NLpALCynBiapplQPlfJiycvWK8wG_zpLVKwyIV59ohsDoHFRbIAg1awc4eDASuDERxI">
Google Optimize isn&rsquo;t the only product to make the visual editor the center of their experimentation experience - Optimizely, VWO, and others have also incorporated visual editing tools into their platforms.
Here at Causal, though, we very deliberately have not.
While visual editors make it easy for non-technical users to make simple changes to copy, color, and content,&nbsp;they come with significant downsides around performance, user experience, and engineering overhead.&nbsp;
As you consider alternatives to Google Optimize, we suggest jettisoning visual editors as a requirement, in favor of deeply configurable features that give you much more control and flexibility.
<h3>How Visual Editors Work: JavaScript Injection</h3>
To understand the downsides of visual editors, it&rsquo;s first important to understand how they are implemented.
Visual editors like Google Optimize 360, Optimizely, and VWO inject JavaScript into your site to display different variations to users.&nbsp;
When a user visits a website that has experiments enabled via a visual editor, the frontend first loads your code; then, it reaches out to the visual editor to obtain additional JavaScript code.
This additional code is then "injected" into the page, modifying its visual appearance or functionality. This can be used, for instance, to perform A/B testing, personalize user experiences, or implement other types of dynamic adjustments to the site.
<h3>The Problems with Visual Editors</h3>
There are some obvious ramifications of this approach to A/B testing.
<h4>Performance Hits&nbsp;</h4>
&ldquo;We're running a redirect test on our homepage and it's caused our mobile page speed to go from low 90s to high 60s, and our desktop speed to go from 99 to 90.&rdquo;
<ul>
<li>
<a href="https://www.reddit.com/r/analytics/comments/ey6tzd/google_optimize_redirect_test_slowing_pagespeed/">r/analytics</a>
</li>
</ul>
&ldquo;If your FCP and LCP warning lights go off, it&rsquo;s time to evaluate your critical rendering path. Something&rsquo;s probably gumming it up. When we first investigated the issue, we had 4 render blocking assets:
&hellip;When we got to our A/B testing script, things cleared up.
<img height="55" width="624" src="https://lh3.googleusercontent.com/Ew_gE31b1adnYJSu69HAxBuid1AJUeBXJoZ6dCtrNAWDth2UyGDWF5fSXcrMIZbgY_VTXA0x9U5yfbLzD63C3-r-M2Hy_rJmRicjpZRNkGyU-3oo1PkSDBzBF4Bc_PmY6JJwVLuegaQVHNFfxbUZ59s" alt="Timeline showing REI.com starts loading at 3.5 seconds after blocking the A B testing script.">
An FCP change from 8 second to 3.5 seconds is a massive improvement. Why does it make such a huge impact?&rdquo;
<ul>
<li>
Casey Carroll at REI
</li>
</ul>
Visual editors can slow sites down by 10% or more. Calling out to an external service, obtaining new JavaScript, and injecting it into the page introduces additional latency. The operation involves a round-trip request over the network, which adds to the load time of the website, hurting conversion and the user experience.&nbsp;
Moreover, it&rsquo;s not just the pages that you are running experiments on that will be affected - all pages where the Google Optimize snippet is installed will take a performance hit.
<h4>Visual &ldquo;Flicker&rdquo; and Poor UX</h4>
The injected JavaScript causes the page to visibly change after it&rsquo;s already loaded, creating a jarring experience for users. In the industry, this is often referred to as &ldquo;the flicker.&rdquo; This is an issue even for simple experiments on copy or color that visual editors are supposedly ideal for.
Companies have&nbsp;<a href="https://www.tezify.com/post/how-we-overcame-the-speed-impact-of-anti-flicker-snippets/">spent</a> <a href="https://engineering.rei.com/web-performance/catch-the-wheel.html">many</a> engineering hours tackling the flicker, and have come up with all sorts of hacky workarounds, all of which are suboptimal. This includes:
<ul>
<li>
Avoiding testing anything &ldquo;above the fold,&rdquo; so that nothing the user immediately sees will flicker. But not experimenting on key parts of your site severely limits the value of experimentation.
</li>
<li>
Testing on only small segments instead of doing A/B, to control how many users have lower performance. But this is likely to result in underpowered experiments.
</li>
<li>
Adding an <a href="https://support.google.com/optimize/answer/7100284?hl=en">&ldquo;anti-flicker&rdquo; snippet</a> that will keep the page content invisible until the injected JavaScript is fully loaded. But this has the side effect of adding significant latency to all your page loads. In general, there&rsquo;s a tension between flicker and performance - if you give Google Optimize more time to load before your page is visible, so the user doesn&rsquo;t experience any flicker, you increase the latency.
</li>
</ul>
<h4>&ldquo;Action at a Distance&rdquo; and Confusing Engineering&nbsp;</h4>
Perhaps the most painful problem is that when engineering teams go to make changes or investigate issues, the logic of what&rsquo;s actually running on the site is obscured by all the injected JavaScript from the visual editor. The codebase itself does not represent an accurate view of what is running in production. This makes it easy to introduce bugs since you might make a change to something and not realize that there is code injection, inducing an unexpected interaction.
Finally, since there are no feature flags in the codebase itself, to debug, you&rsquo;ll have to go into the visual editor and manually check what experiments are running. So unless the team is very disciplined about cleaning things up, there&rsquo;s always going to be a weird accrual of information that only lives in the visual editor.
<h4>The Benefits of Visual Editors</h4>
Visual editors do have some benefits, which is why they&rsquo;ve been popular. They make basic experimentation very easy for non-technical users by providing an intuitive visual interface to create tests, and they require almost no engineering resources to set up initial experiments. This allows teams to get quick wins and generate results to justify further engineering investment.
<h4>A Better Option: Causal&rsquo;s Deep Feature Flagging</h4>
Causal offers an alternative that provides the key benefits of visual editors without the significant downsides. Our deep feature flagging solution gives business users an easy way to configure feature flags and set up experiments through our UI. But under the hood, we integrate directly into your codebase rather than using JavaScript injection.
This means:
<ul>
<li>
Faster page loads. While our product also loads JavaScript, the performance impact is less significant due to the smaller file sizes and lower setup latency of our scripts. We also support server-side rendering to minimize latency.
</li>
<li>
No visual flicker or poor UX. Your users see a single, consistent experience because we load variations directly rather than layer them on after the initial load. Even if you are not using server-side rendering, the UI will never change from one thing to another. Instead, it will just display a loading state.
</li>
<li>
Easy debugging for engineers. The logic for what&rsquo;s running in production lives in your codebase, not in a separate visual editor tool, so engineers can reason about the system easily.
</li>
</ul>
While Causal may require slightly more work to get started for the first experiment, the lack of accumulating &ldquo;code debt&rdquo; and additional maintenance overhead means lower costs over the lifetime of a program. Causal unlocks high-velocity experimentation without excess burden on engineering.
<h3>Conclusion</h3>
Visual editors solve an important problem, allowing non-technical users to make basic changes and run simple tests. But for any serious experimentation program focused on performance, user experience, and scale, configurable features is a far better approach. Causal offers the flexibility that visual editors promise without the messy realities that come along with them.

Introduction Google recently announced they are sunsetting Google Optimize 360, their no-code A/B testing tool. While Google Optimize was po

Why visual editors are the wrong approach to A/B testing

Introduction Google recently announced they are sunsetting Google Optimize 360, their no-code A/B testing tool. While Google Optimize was popular and Google should get credit as the one major tech player to offer an A/B testing platform with a free plan, there’s one aspect of Google Optimize’s design that we ...

Causal Blog