MATLAB

The Story of TIMEIT

The function timeit started out in an engineering textbook about image processing using MATLAB.

Steve Eddins

15 Jan 2025 • 8 min read

Mike Croucher published a blog post last spring that prompted a long comment thread about the MATLAB function timeit. It caught my eye, and I followed (and contributed to) the thread.

In case you don’t know about timeit, this function in intended to provide an easy and reliable way to measure the execution speed of a bit of MATLAB code. For example, how long does it take to compute the 2-D FFT of a 1536x2048 matrix? First, you make a function handle that performs exactly that operation (on precomputed data), and then you pass that function handle to timeit.

A = rand(1536,2048);
f = @() fft2(A);
timeit(f)


ans = 

    0.0100

It takes about 10 ms (on my 2021 M1 MacBook Pro).

Today I’d like to tell the story of timeit: its beginning as part of a textbook, its design, how and why it works the way it does, and how it came to be in MATLAB. I’ll also mention a couple of recurring complaints about the function, along with some ideas about resolving those complaints. I will suggest possible changes and even recommend how to ask MathWorks for them. (Note: It does no good to ask me for changes, as I retired from MathWorks last spring.)

TIMEIT Started in a Textbook

The function timeit started out in an engineering textbook, not in MATLAB itself. When I was working on the 2nd edition of Digital Image Processing Using MATLAB, I saw that my coauthors had written several examples that measured the execution times of MATLAB code. Early drafts of these examples looked something like this:

tic
B = fft2(rand(1536,2048));
toc

As a MathWorks toolbox developer, I was familiar with the pitfalls of that method, such as:

First-time effects

When functions are first called in a MATLAB session, additional time is needed to load those functions into memory and parse them. Typical workaround: Execute the code being measured several times before timing it.

System noise

A modern computer is always actively doing things in the background, such as responding to network traffic, downloading email messages, updating a search index, or syncing cloud storage locations. At any given moment, a burst of such activity could temporarily delay the MATLAB code being executed. Typical workaround: Repeat the timing experiment many times, usually with a for-loop, and mitigate the effect of outliers by reporting the median or minimum result from the experiments.

Unintentional code inclusion

The performance measurement above is measuring the execution time of rand, not just fft2. Typical workaround: precompute any values needed and store them in variables to be used by the code being measured.

Script vs function effects

MATLAB often executes code much faster if it is in a function, as opposed to code in a script or entered directly in the Command Window or Editor. Typical workaround: put the code to be measured into a function file.

Measurement overhead

Timing functions, such as tic and toc, for-loops, function calls, and other measurement machine can affect the result. Typical workaround (partial): Execute the code being measured many times within one pair of calls to tic and toc.

That’s a lot to deal with! It seemed that performing high-quality performance measurements in the textbook examples might require a lot of awkward, distracting, hard-to-read code. To avoid that, I began working on a measurement function that would be easy to use and that would automatically incorporate measurement best practices. Along the way, I consulted with Bill McKeeman, a MathWorker (now retired) with decades of experience with programming languages and associated technology. He had been doing some performance benchmarking work in MATLAB, so he had a lot of good advice to share. See his whitepaper on the File Exchange.

How TIMEIT Works

The result of my efforts was the function timeit. This function was submitted to the File Exchange in 2008 and included with Digital Image Processing Using MATLAB in 2009. See my 28-Mar-2008 MATLAB Central blog post, “TIMEIT Benchmarking Function”

Here is what timeit does. The goal of this procedure is to produce a reliably consistent measurement that meaningfully reflects, and to do so with a simple interface.

Warm up tic and toc by calling them several times

Compute a first, rough estimate of execution time

This is done by running the function handle in while loop until at least a millisecond has passed. This step also serves to warm up the code being measured.

Use the rough estimate to compute the number inner-loop repetitions

The inner loop will be the loop that is directly timed by a pair of calls to tic and toc. The inner loop is constructed to run for at least a millisecond.

Use an outer loop to execute the inner loop 11 times

You can think of this as performing the core timing experiment 11 times. If running the inner loop 11 times is expected to take longer than 15 seconds, then the number of outer loop repetitions may be reduced, but it will always be repeated at least 3 times.

Compute the median outer loop execution time

Dividing the median outer loop execution time by the number of inner loop repetitions gives the almost-final result.

Compensate for measuring overhead

To compute the final result, adjust for estimated function handle and tic/toc call overhead.

Going into MATLAB

After several years of experience with the timeit function as distributed on the File Exchange, several of us in MATLAB developmebt had gained confidence in the function and its value, and we thought that it, or something similar, should go into MATLAB.

So, sometime around 2012, a MATLAB developer, Campion, took on the project of getting timeit into MATLAB. He validated the design and implementation, checking with core MATLAB language developers that it was still sound. He successfully took the design through MATLAB design review. And then he integrated the function into MATLAB, working with a quality engineer and writer to make sure everything was working correctly and was documented clearly.

The function timeit shipped with MATLAB for the first time in R2013b. See my 30-Sep-2013 MATLAB Central blog post, “timeit makes it into MATLAB.” A few years later, I removed the version on the File Exchange.

Issues with TIMEIT

The function timeit has accumulated its share of complaints over the years. In Mike’s blog post, he explained why he doesn’t like it:

I don't feel like I get more for my effort [with timeit]. tic/toc gives me a number. timeit gives me a number. It may be 'more robust' but I have no sense of how much more robust it is. How many times was it run and why can't I control it? What was the maximum run time and what was the minimum? In short, I want some evidence for the multiple runs that the documentation tells me have taken place and I want to be able to control the number of runs.

Others in the comment thread also wanted more control over how the experiments are executed and to have more experiment details returned as output.

Some (such as Royi) think that it would have been better to return a result on the minimum experiment time instead of the median time, or that the user should be able to choose the desired computation method. Blog reader goc3 had the most detailed set of requests about outputs and optional inputs.

Some questioned why timeit does not determine the number of output arguments automatically.

Also, within MathWorks, we have noticed that needing to understand anonymous functions is a impediment to many users. This is a MATLAB language feature that not everyone understands well. That conceptual issue didn’t come up in the comment thread. I speculate that the reason is that most people who comment on MATLAB blog posts are MATLAB power users who understand anonymous functions, at least to some degree.

I can provide some additional context about these questions here. First, I want to mention a key design goal: I wanted timeit to automatically apply a broad set of performance measurement best practices with an interface that was very simple and that required no specialized knowledge about performance measurement. That’s the main reason timeit has such a bare bones interface.

About automatic determination of the number of output arguments, I said this in Mike’s blog comment thread:

The number of outputs of a function cannot be determined reliably enough to do this. In addition to the issue of multiple function behaviors that vary based on the function's inputs, which has already been mentioned, nargout returns -1 for anonymous function handles, which is a very common scenario for timeit. The function nargout also returns -1 for any function that has been implemented using varargout.

For these reasons, I rejected the possibility of automatic determination of the number of output arguments for the original timeit design. When the MATLAB development team later evaluated timeit for inclusion in MATLAB, this question was revisited, and the same conclusion was reached.

About returning the median execution time instead of the minimum time, I said:

This is something that I discussed several times with Bill McKeeman, who worked at MathWorks for a while before his retirement. In Bill’s white paper on performance measurement in MATLAB (File Exchange link), he discusses the choice between minimum and median. If the purpose is to improve code, then using the minimum is recommended. If the purpose is “to measure user experience,” Bill says, then “median time is the more reasonable metric.”

To Bill’s comments, I would add my observation that it often requires substantially fewer repetitions, and therefore much less time, to obtain “repeatable results” using the median than using the minimum. In my original work, sometime around 2008, I experimented using both metrics. In typical cases involving functions whose execution times was not especially sensitive to system effects, the two metrics were usually very close.

In situations where the two metrics were different, the minimum value might appear only once, for example, among ten runs. When you then ran the experiment a second time, the minimum value would often be different.

In those scenarios, then, the minimum value metric would not be representative of typical user experience, and it would also take many more runs to get a reliably repeatable number.

Performance Testing Framework

When considering issues with timeit, it is helpful to know that MathWorks introduced an entire performance testing framework in MATLAB in R2016a. You can create performance tests based on a script or a function, or you can create more sophisticated tests based on the class matlab.perftest.TestCase. You can run different types of time-based experiments. The framework offers a lot of control over the experiments, and it returns an extensive set of experiment data and statistics. See “Overview of Performance Testing Framework.” for much more information.

When considering possible changes to timeit, one should be aware of the capabilities that already exist in the performance testing framework.

Possible Changes

Well, of course it is natural that I like timeit, because I designed it. (big smile emoji)

But really, there is a lot good feedback out there, and there are many possible changes that could be made. To prompt your thinking, here are some design ideas that came to me as I was writing this post:

In the performance testing framework, extend the runperf syntax to allow runperf(f), where f is a function handle. With that change, a call to timeit could be replaced by a call to runperf with no other change, and the user would get back more information and have more control.
Give timeit a second output argument, which might be a structure containing experiment data and statistics.
Give timeit some optional named arguments to control various aspects of its behavior.
When timeit is called with no output arguments, display a formatted summary of experiments to the Command Window along with the returned time. There is ample MATLAB design precedent for this particular kind of what I call nargout == 0 behavior.

Are you interested in any of these changes? Or do you have your own ideas? Well, don’t try to convince me about them! If you’ve forgotten, I’m retired.

Instead, let MathWorks know! Submit a support case with an enhancement request. Or post on Answers or Discussions.

I have created a MATLAB Central Discussions thread to serve as a place for followup discussions about this post and about timeit. If you are interested, I encourage you to participate in the discussion there. The commenting system here doesn’t work all that well, and it will take me a while to get around to investigating alternatives. The Discussions forum will be better.

Pro Tip for Submitting Enhancement Requests

When you submit an enhancement request to MathWorks, don’t stop with just a description of a particular functional design (new function, new syntax, new controls in an app, etc.). Include also a description of specifically how the new or revised capability would make a difference in your day-to-day work. Is there some task you are trying to perform that is currently awkward or difficult? Let MathWorks know about this broader task and how your suggestion fits in. It helps them prioritize their work, and it helps them to be sure that a new design will really address your needs. I know from years of personal experience that those workflow descriptions really do get read, and they show up later in design specs as motivation and illustrative examples.