<h1>The Future of Software Development</h1>
<p>Most experienced software engineers will recall constantly needing to stay up-to-date with changes in software technologies in order to partake in most of the industry. Today&#39;s landscape is not an exception, but it does come with some notable differences.</p>
<p>The new players in the field (as it relates to this blog post) are not new programming languages, libraries, nor philosophies of code quality. Instead, they are large language models and their ability to code, test, make architectural decisions, and (soon) automatically perform deployment and other necessary actions.</p>
<h2>Coding and Testing</h2>
<p>In any given timeframe, software engineers are now able to create vastly more value, as some of the most time-consuming parts of coding are now essentially automated. Tools like GitHub Copilot, Cursor, and more are able to quickly implement:</p>
<ul>
<li>Boilerplate solutions</li>
<li>Code Refactors</li>
<li>Translation to alternative programming languages</li>
<li>Debugging</li>
<li>Unit test creation</li>
<li>Increasingly, entire features across multiple files</li>
</ul>
<p>This allows mindful software engineers to move quickly, focusing their efforts on:</p>
<ol>
<li>Directing the AI</li>
<li>Making high-level and architectural decisions (in order to do (1))</li>
<li>Carefully reviewing AI output</li>
</ol>
<p>At Vectorial, we take this very, very seriously. Each of our engineers is highly encouraged to take advantage of models and tools such as <a href="https://openai.com/index/openai-o1-system-card/">OpenAI&#39;s o1</a>, <a href="https://www.anthropic.com/news/claude-3-5-sonnet">Anthropic&#39;s Claude 3.5 Sonnet</a>, <a href="https://www.cursor.com/">Cursor</a>, and/or <a href="https://github.com/features/copilot">GitHub Copilot</a>. However, we recognize that these tools often make mistakes, so we also emphasize the engineers&#39; responsibility to review code heavily, and require all delivered code to be peer-reviewed by another human engineer.</p>
<p>The balance we strike here allows us to deliver software and AI solutions at a higher quality <em>and</em> a greater speed than ever previously possible.</p>
<h2>Architectural Decisions</h2>
<p>State-of-the-art models are increasingly becoming useful at more high-level tasks, such as designing entire technical architectures. Models such as o1 and Claude 3.5 Sonnet can often create exceptional first drafts of an architecture for a new product or feature.</p>
<p>These architectures are still prone to mistakes, which have the potential to be more dangerous than code bugs, as they are less detectable and fixable. Following them blindly can often be a one-way door that you can&#39;t easily reverse down the line.</p>
<p>That being said, having a human engineer treat these drafts as mere (possibly faulty) suggestions is often higher-quality and discovered faster than the architectures that a junior or mid-level engineer would come up with on their own.</p>
<h2>Automated Software Engineering</h2>
<p>As frontier AI labs continue to improve these LLMs, we continue to progress toward a future in which these AI models are able to perform every aspect of a modern software engineer&#39;s job. Currently, it does not seem clear to us at Vectorial whether or not software engineers&#39; skills will continue to be useful by &quot;moving up the abstraction ladder&quot; if non-technical people can direct the AIs just as effectively.</p>
<p>Products or technologies like <a href="https://devin.ai/">Devin</a>, <a href="https://github.com/All-Hands-AI/OpenHands/">OpenHands</a>, <a href="https://aider.chat/">Aider</a>, <a href="https://v0.dev/">v0</a>, and <a href="https://www.cursor.com/features">Cursor Agent</a> are all early versions of this. You simply explain in natural words what you want to build and the models continuously run, working on the code, and eventually come back with either questions or results (or failures, hence the &quot;early&quot; label!).</p>
<h2>An Example</h2>
<p>Take a quick look over our website. Look at the particle effects, the landing page, the blog post you&#39;re currently reading (not the words, but the code). This is all nearly 100% created by Claude 3.5 Sonnet (using Cursor) and o1. The particle effects are a custom implementation, fully developed and iterated on by o1. We estimate that building the website to its current state would have taken at least 3-5x as many developer hours without the use of AI. A completely non-technical person could have likely built and deployed this website in just 2-3 days.</p>
<video autoplay loop muted playsinline class="w-full h-auto rounded-lg my-4" id="particle-video">
  <source src="/images/vectorial-site-particles.mp4" type="video/mp4">
  Your browser does not support the video tag.
</video>

<script>
  document.addEventListener('DOMContentLoaded', () => {
    const video = document.getElementById('particle-video');
    const observer = new IntersectionObserver((entries) => {
      entries.forEach(entry => {
        if (entry.isIntersecting) {
          video.play();
        } else {
          video.pause();
        }
      });
    }, { threshold: 0.6 }); // Start playing when at least 60% of the video is visible
    
    observer.observe(video);
  });
</script>

<h2>How to Prepare</h2>
<p>Today, it is a good idea for software engineers to get acquainted with generative AI coding tools and agents. It is common to have a few bad initial experiences with generative AI, and avoid using it entirely, to one&#39;s own detriment. Instead, one can refine their abilities of using the tool (similar to the older concept of &quot;<a href="https://en.wiktionary.org/wiki/Google-fu">Google-fu</a>&quot;) and can remember to push the limits of the newest models as they are released, as previously failed uses may now be within the new models&#39; capabilites. It is also important, for the time being, to maintain existing programming skills, to avoid helplessness in times without access (such as in jobs/interviews where generative AI usage is forbidden).</p>
<p>In the medium-term, software engineers should prepare for the likely case of rapid change followed by near-full automation. This involves the job requirements changing rapidly every year or even every few months. Soon after, it may become the case in <a href="https://www.vectorial.us/blog/short-term-ai-progress">the late 2020s</a> that non-technical people can use these AIs just as effectively as software engineers. In this scenario, it&#39;s best to save as much of your income as possible today, as it may drop significantly (or to zero).</p>
<p>In this same scenario, software businesses should also prepare. &quot;Tech debt&quot; that can stay &quot;unpaid&quot; for a couple years may be worth making, since the cost of fixing it will continue to drop. It may make sense to stop hiring junior software engineers, and eventually mid-level engineers.</p>
<p>And businesses in general can prepare for significant technological disruption. For example, modern CRMs can, at risk of heavily oversimplifying, be described as a large user interface and user experience (UI/UX) operating on a database. Will that paradigm remain the same if a small team is able to simply ask OpenAI&#39;s future o4 or o5 &quot;generate and deploy code that can listen in on users&#39; calls, emails, and messages, and store data related to deals, meetings, etc. for them to ask about later&quot;?</p>
<h2>Key Takeaways for Software Engineers:</h2>
<ul>
<li>Adopt AI tools to stay competitive, but maintain rigorous review processes.</li>
<li>Keep your skills sharp — both technical (for deeper debugging/coding/architecture) and in AI usage.</li>
<li>Prepare for rapid changes in the job market: be financially cautious and professionally adaptable.</li>
</ul>


The Future of Software Development

<h2>AI Progress So Far</h2>
<p>The past few years have seen unprecedented advances in artificial intelligence, particularly through the usage of transformers in autoregressive and diffusion architectures. Since the release of GPT-2 in 2019, we&#39;ve witnessed a cascade of increasingly capable AI models from multiple organizations. These models have progressed from basic text completion to complex reasoning, coding, and even multimodal capabilities combining text, images, video, and audio.</p>
<p>Key milestones include:</p>
<ul>
<li>2019:<ul>
<li>Demonstrated the success of self-supervised training for decoder-only transformers (<a href="https://openai.com/index/better-language-models/">GPT-2</a>)</li>
</ul>
</li>
<li>2020:<ul>
<li>Set new standards for language understanding/generation; demonstrated in-context learning (<a href="https://openai.com/index/language-models-are-few-shot-learners/">GPT-3</a>)</li>
<li>Demonstrated breakthrough protein structure prediction (<a href="https://deepmind.google/discover/blog/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology/">AlphaFold 2</a>)</li>
<li>Public launch of driverless taxis in geo-fenced areas (<a href="https://waymo.com/blog/2020/10/waymo-is-opening-its-fully-driverless-service-in-phoenix/">Waymo</a>)</li>
</ul>
</li>
<li>2021:<ul>
<li>First major product release of large language models (<a href="https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/">GitHub Copilot</a> and OpenAI&#39;s <a href="https://openai.com/index/evaluating-large-language-models-trained-on-code/">Codex</a>)</li>
</ul>
</li>
<li>2022:<ul>
<li>Impressive multimodal understanding by processing text and image data jointly (<a href="https://deepmind.google/discover/blog/tackling-multiple-tasks-with-a-single-visual-language-model/">Flamingo</a>)</li>
<li>Compute-optimal scaling laws for transformer large language models (<a href="https://deepmind.google/discover/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/">Chinchilla</a>)</li>
<li>Significant advancement in text-to-image generation (<a href="https://openai.com/index/dall-e-2/">DALLE-2</a>, <a href="https://imagen.research.google/">Imagen</a>)</li>
<li>Brought conversational AI to the mainstream (<a href="https://openai.com/index/chatgpt/">ChatGPT and GPT 3.5</a>)</li>
</ul>
</li>
<li>2023:<ul>
<li>Massive improvements in reasoning and multimodal capabilities (<a href="https://openai.com/index/gpt-4-research/">GPT-4</a>)</li>
</ul>
</li>
<li>2024:<ul>
<li>Expanded multimodal capabilities, combining text, image, video, and audio input (and output for text and image) into the same models (<a href="https://openai.com/index/hello-gpt-4o/">GPT-4o</a>)</li>
<li>Heavy improvements in reasoning and overall model quality (<a href="https://www.anthropic.com/news/claude-3-5-sonnet">Claude 3.5 Sonnet</a>, <a href="https://openai.com/index/learning-to-reason-with-llms/">o1</a>)</li>
<li>Impressive video generation models (Sora, Veo 2, Runway, Kling)</li>
<li>Impressive music generation models (Suno, Udio)</li>
<li>Massive improvements in structure prediction of biomolecular interactions (<a href="https://blog.google/technology/ai/google-deepmind-isomorphic-alphafold-3-ai-model/">AlphaFold 3</a>)</li>
<li>Significant improvements for driverless cars <em>without</em> geo-fencing (Tesla FSD v12 and v13)</li>
</ul>
</li>
</ul>
<h2>Have LLMs hit a wall?</h2>
<p>At first glance, after going from GPT-2 -&gt; GPT-3 -&gt; GPT-4, it may feel like progress has stalled. However, we think this is the common pitfall associated with desensitization or <a href="https://psychcentral.com/health/hedonic-adaption">hedonic adaptation</a>. We believe there are three major sources of this increased desensitization:</p>
<ol>
<li>People who were previously paying attention are now paying attention <em>much more</em> now, so AI progress feels slower. Recall that it took almost 3 years from the GPT-3 announcement until the GPT-4 announcement. As of this writing, it has not even been 2 years since the GPT-4 announcement.</li>
<li>Model releases are much, much more frequent. This stems from more frontier AI labs (OpenAI, Anthropic, Google Deepmind, Meta), more secondary AI labs (Deepseek, Qwen, Mistral), and a shorter release cycle from each. Thus, there were much fewer models between GPT-3 and GPT-4 than between GPT-4 and GPT-4o</li>
<li>AI models are increasingly saturating both formalized benchmarks and unformalized ones (i.e. &quot;vibes&quot;). As models improve, people resort to harder and harder questions, and thus the rate of &quot;failure&quot; in real-life usage stays relatively constant from 2 years ago.</li>
</ol>
<p>The result of these sources is that AI progress can feel slower than it was 2 years ago, or 4 years ago. However, in reality, the models we have access to today are significantly better than models from 1-2 years ago, such as GPT-4 and GPT-4 Turbo. This is evident in essentially every single benchmark, as well as in anecdotal experience from our team.</p>
<h2>Will LLMs hit a wall?</h2>
<p>Assuming LLMs have not yet hit a wall: will they do so in the near future? Common arguments for &quot;yes&quot; include: running out of data, fundamental limits of autoregressive transformers, and diminishing returns of more/better data and more compute. Let&#39;s visit each point directly.</p>
<ol>
<li><p><strong>Running out of Data</strong></p>
<ul>
<li>Llama 3, the best knowledge we have of modern state-of-the-art LLM training, used 15 trillion tokens.<ul>
<li>Most of this data is from mass internet web crawls.</li>
<li>Some estimates put <a href="https://en.wikipedia.org/wiki/Books_published_per_country_per_year">the number of books published each year</a> at ~2.2 million. Assuming the average number of words per book is 80,000, that&#39;s ~235B tokens (~175B words) published every year. That amounts to 14 trillion high-quality tokens since 1960, matching the amount Llama 3 was trained on.</li>
<li>Google&#39;s YouTube has about <strong>1 million hours of footage uploaded every day</strong>. At 30 frames per second, and <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video-understanding">258 tokens per frame</a>, that&#39;s about 27 trillion tokens of video added every single day. That&#39;s about <strong>one quadrillion</strong> tokens for every year of YouTube. This even ignores audio tokens from the videos as well! The tens of trillions of tokens used to train today&#39;s state-of-the-art models pale in comparison.<ul>
<li>Keep in mind, this is just one product. There are similarly crazy numbers of human-generated tokens on other social media services, such as Facebook, Instagram, LinkedIn, TikTok, Twitter, Reddit etc.</li>
</ul>
</li>
</ul>
</li>
<li>Frontier AI labs are finding surprising success with synthetic data approaches.<ul>
<li>For example, generating code or math using LLMs, and, using automated deterministic tests or even LLM judges, choosing high quality responses and performing Reinforcement Learning, fine-tuning, or even pretraining on them.<ul>
<li><a href="https://x.com/polynoamial/status/1834280425457426689">OpenAI has announced</a> that they used this methodology to create o1</li>
<li>Many other major and small labs have claimed they&#39;re seeing success with synthetic data.</li>
</ul>
</li>
</ul>
</li>
<li>In any case, if we really do run out of data, AI scientists have shown that scaling up model size results in better sample efficiency. We are currently in an overtraining paradigm (due to outsized inference costs compared to training costs), but if we transition to being constrained by data, there is a lot of room for improvement by entering an undertrained paradigm (by massively scaling model sizes).</li>
</ul>
</li>
<li><p><strong>Fundamental Limits of Autoregressive Transformers</strong></p>
<ul>
<li><a href="https://arxiv.org/pdf/2411.01992">Recent research</a> has mathematically proven that autoregressive, decoder-only transformers are <a href="https://en.wikipedia.org/wiki/Turing_completeness">Turing-complete</a> if given the correct input</li>
<li>Time and time again, skeptics of the modern generative AI paradigm have predicted that these models would fail at specific tasks, and time and time again they have quickly improved enough to succeed at those tasks.</li>
</ul>
</li>
<li><p><strong>Diminishing Returns of Increasing Data Size, Data Quality, and Compute</strong></p>
<ul>
<li>Of the three, this argument has the best potential to be right. We can never be sure that a certain trend will continue, and this is no different. However, we can take hints from the frontier labs, who have at least a few months headstart on seeing these models in action.<ul>
<li>Noam Brown (OpenAI multi-agent researcher): &quot;<em>We announced @OpenAI o1 just 3 months ago. Today, we announced o3. We have every reason to believe this trajectory will continue.</em>&quot; (<a href="https://x.com/polynoamial/status/1870172996650053653">link</a>)</li>
<li>Dario Amodei (Founder of Anthropic) loosely predicts 2026 or 2027 for &quot;very powerful AI&quot; (<a href="https://x.com/tsarnick/status/1856065476759302167">link</a>)</li>
<li>Sam Altman (Founder of OpenAI): &quot;<em>We are now confident we know how to build AGI as we have traditionally understood it. We believe that, in 2025, we may see the first AI agents “join the workforce” and materially change the output of companies.</em>&quot; (<a href="https://blog.samaltman.com/reflections">link</a>)</li>
</ul>
</li>
</ul>
</li>
</ol>
<h2>What might 2030 look like?</h2>
<p>There are a few major trends to pay attention to:</p>
<ol>
<li>More data and more high-quality data are being used to <a href="https://www.clrn.org/what-is-pretraining-and-post-training/">pretrain and post-train</a> these models.</li>
<li>Progressively increasing compute.<ul>
<li>Massively parallelized computer chips are continuing to improve in speed, memory, and other metrics.</li>
<li>Larger and larger clusters of these chips are being built constantly. <a href="https://blogs.oracle.com/cloud-infrastructure/post/now-ga-largest-ai-supercomputer-oci-nvidia-h200">OCI Supercluster</a> from Oracle, <a href="https://www.tomshardware.com/desktops/servers/first-in-depth-look-at-elon-musks-100-000-gpu-ai-cluster-xai-colossus-reveals-its-secrets">Colossus</a> from xAI, and rumors of <a href="https://www.tomsguide.com/ai/meet-stargate-the-dollar100-billion-ai-supercomputer-being-built-by-microsoft-and-openai">Stargate</a> from OpenAI and Microsoft</li>
</ul>
</li>
<li>Increasing focus and efforts on post-training, using methods such as RLHF, RLAIF, supervised fine-tuning, and other reinforcement learning and synthetic data methods based on provably validity of response.</li>
<li>Improving on the vanilla transformer architecture. Papers like the <a href="https://arxiv.org/abs/2412.09871">Byte-Latent Transformer</a>, <a href="https://sakana.ai/transformer-squared/">Transformer^2</a>, <a href="https://arxiv.org/abs/2412.06769">Reasoning in Continuous Latent Space</a>, and many others are recent, seemingly successful, attempts at this.</li>
<li>Distributed training. As cluster sizes hit physical and infrastructure limits, research is being done on the potential for distributed training of models. <a href="https://www.primeintellect.ai/blog/intellect-1">Prime Intellect&#39;s DiLoCo</a> and <a href="https://github.com/NousResearch/DisTrO">Nous Research&#39;s DisTrO</a> are examples of this.</li>
</ol>
<p>If we extrapolate these trends, 2029 and 2030 may look like enormous (even relative to today), distributed training runs for heavily multimodal (text, image, video, and audio input <em>and</em> output) models. These models may be able to autonomously perform long-horizon tasks, such as: building complex web applications, creating detailed visual media, managing a team of and writing detailed research on computer science, artificial intelligence, mathematics, physics, and more (chemistry and biology may require more human intervention due to laboratory tasks).</p>
<p>This would be the beginning of massive job automation, where most white-collar jobs would be able to be fully replaced by AI. Depending on the manufacturing speed of general-purpose (e.g. humanoid) robots, we may start to see significant blue-collar job replacement as well.</p>
<p>It would also be the beginning of a lack of scarcity (relative to today&#39;s society). After controlling for potential governmental increases of monetary supply, prices for nearly every good or service should drop significantly, as the cost of producing them drops, and as overall efficiency increases.</p>


Blog

The Future of Software Development

AI Progress in the Second Half of the 2020s