Lecture 2: Data Collection Techniques and Program Design

• Download English-US transcript (PDF)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make a donation or to view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

GABRIEL SANCHEZ-MARTINEZ: Any questions on Homework 1 before we get started?

AUDIENCE: Yeah.

GABRIEL SANCHEZ-MARTINEZ: OK, fire away.

AUDIENCE: I guess, first, do you think we have like this minimum cycle time, like a theoretical minimum cycle time and then what was actually [INAUDIBLE] cycle time?

GABRIEL SANCHEZ-MARTINEZ: So cycle time, just to review-- it's the time that it takes a bus to-- from the time [AUDIO OUT] for a trip. It goes all the way one way, has to wait at the other end to recover the schedule, comes back, waits to recover, and is ready to begin the next round. So that's a cycle.

AUDIENCE: Since you have [INAUDIBLE] going on, if you had 4.1 buses, then you use a cycle time. Then obviously, you can't do that?

[INTERPOSING VOICES]

GABRIEL SANCHEZ-MARTINEZ: So you would need five buses--

AUDIENCE: Yeah.

GABRIEL SANCHEZ-MARTINEZ: --if that's what you've got. Or you would have to do a trade-off with reliability if that were to happen.

AUDIENCE: I think most of my questions were on this very last couple of questions.

GABRIEL SANCHEZ-MARTINEZ: Yeah.

AUDIENCE: We were aggregating a bunch of data for-- [INAUDIBLE] you did it across both directions and then asked, how does it change when you would like to evaluate each direction separately in layover time?

GABRIEL SANCHEZ-MARTINEZ: This is the penultimate question, correct?

AUDIENCE: Yeah.

GABRIEL SANCHEZ-MARTINEZ: So that's the hardest question on the assignment.

AUDIENCE: OK.

GABRIEL SANCHEZ-MARTINEZ: It is a challenge question because there are different cases that you have to analyze. That's maybe the hint, right? There are some cases. And for each case, there is a probability that that case will occur.

AUDIENCE: Yeah.

GABRIEL SANCHEZ-MARTINEZ: And-- let's see if this starts-- there's a probability that it will occur and then a consequence, or something happens in that case. So you have to look at each case and then aggregate the cases together, if that make sense.

AUDIENCE: Yes.

GABRIEL SANCHEZ-MARTINEZ: We're taking questions for Assignment 1, which is due on Thursday. Any other questions?

AUDIENCE: That's it.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: It is due at 4:00 so at class time essentially, yeah. I actually [AUDIO OUT] if you 4:00. I said 4:05, so you have five minutes.

AUDIENCE: Can you [INAUDIBLE] what assumptions there are [INAUDIBLE]?

GABRIEL SANCHEZ-MARTINEZ: In what question?

AUDIENCE: When you said it seems to be the reasoning or assumption about the schedule [INAUDIBLE]? Which metric do you use? Based on the data, which [INAUDIBLE]?

GABRIEL SANCHEZ-MARTINEZ: Yeah, so that's Question 3, correct?

AUDIENCE: Yeah.

GABRIEL SANCHEZ-MARTINEZ: So I can't really explain. I can't give you the answer to the question. So what I'm looking for there is your intuition and your understanding of why you would pick which statistics from Question 2, where it tells you calculate all these things. Now I'm saying pick from those statistics what you would use for t and for r. And you may want to combine different statistics for the computation of r. Yeah?

AUDIENCE: [INAUDIBLE] multiple valid responses but--

GABRIEL SANCHEZ-MARTINEZ: Yes, some more valid than others, but some that are definitely invalid and some that are almost 100% valid but not 100% valid. So there are several correct answers, and some that are very good answers because you can justify the choice of the statistic conceptually. Yeah. Any other questions on Homework 1? I can take some more questions after class, if that's OK. So we had a snow day if you had a good time, and/or at least, you could use it to catch up. So the schedule is a little different now. I've posted an update about that on Stellar (class site).

There's a new syllabus. And we're going to do some [AUDIO OUT] different [AUDIO OUT]. You may remember that we have three introductory classes on topics of [INAUDIBLE]. And then, we had model characteristics and roles. And then, [AUDIO OUT]. We're going to shuffle a little bit. [AUDIO OUT] Microphone working? So because the second assignment is on data collection, we're going to cover that today. And we're going to give you that homework today, so that you can get started on the data collection side.

Then, we're going to cover some of the short-range [INAUDIBLE] of planning concepts. Nema is going to do that-- Nema Nassir. You might recall him from the previous lecture. And then, we'll finish with [INAUDIBLE] and costs in March the 2nd, OK? So remember, there's no class on Monday the 21st.

AUDIENCE: You mean Tuesday?

GABRIEL SANCHEZ-MARTINEZ: Sorry, yes, Tuesday. I think, there's no class on Monday. And then, Tuesday there are classes. But it's Monday's schedule. So we don't have class. Thank you for bringing that up. OK. I'll leave Homework 2 for when we finish with the lecture. But I'll distribute it later. So let's just get started on that. So data collection techniques and program design-- that's the topic for today.

Here's the outline. So we're going to cover a summary of current practice quite quickly. Then, we're going to talk about data collection program design process, the needs, the data needs, the techniques for data collection, the sampling. We're going to get into the details of how we get sample slices. And we're going to finish with special considerations for surveys and surveying techniques. so where are we? Where is the transit industry in terms of data collection, and sampling, and these things?

Largely, there's been a transition from manual to automatic data collection. As you might imagine, with the internet of things, and sensors, and the internet, and wireless, it used to be that if you wanted to have statistics on your running times, you had to send people out. We call those people checkers. And those checkers would have notebooks and record running times, and number of people boarding, and these things. Nowadays, with the modern systems, especially the modern systems, we have several sensors and types of sensors that collect some of that data for us. So we're going to cover both approaches.

[INAUDIBLE] data collection to supplement [INAUDIBLE] data collection. And if you happen to be consulting for a developing country that is working with a system that has not yet brought in automatic data collection technologies, it's also useful to know all about the manual design and manual data collection process. [AUDIO OUT] took this class and ended up working in large consulting firms have gone off to help countries put in new transit systems.

And one of the first things they have to do is back to these slides and see what the plan is going to be, and how many people you need, and how much it's going to cost. So very useful topic. So as I said, there's automatic data collection. There's manual data collection. There's sometimes a mix of data collection techniques. Often, what happens is that we just send people out and collect data. Or we just extract a sample of automatically collected data.

And we don't really think about sampling, and the confidence interval, and how sure are we of that result that we're going to influence policy or make decisions that will affect service. How sure are we of those? So statistical validity. Often, there's an efficient use of data. And ADCS, which is Automatic Data Collection Systems-- we'll use that abbreviation throughout the course- presents a major opportunity for strengthening data to support decision making. We'll talk about how that happens. Let's first compare manual and automatic data collection.

So what happens with manual data collection? You hire people, as I said. You hired checkers. So initially, there's no setup cost. There's a low capital cost to that. But there's a high marginal cost because if you want to collect more data, you have to hire more people. Does that make sense? If you want to bring in an automatic data collection system, you might have to retrofit all your buses with AVL sensors. And that's going to cost you initially. So that's a high capital cost relatively. But low marginal cost-- once you have those systems in place, they keep collecting data for you. And it's almost free.

You do need some maintenance on these equipments. But comparing to manual data collection, you have low marginal cost. Because of that marginal cost difference, it tends to happen that when you have manual data collection, you only pay checkers for small sample sizes-- just what you need. Whereas, once you put in automatic data collection systems, they keep collecting data. So you get much bigger data. Bless you. OK, in both cases, we can collect data and analyze it for aggregate analysis and disaggregate analysis.

So you might want passenger-specific data on things. Or you might want things like just averages and aggregate things, total number of passengers using the system. And when you're doing manual data collection, you can look at quantitative things, things you can measure and count. Or you can also observe things qualitatively. One example that I saw in a recent paper was considering the [? therivation ?] by student in some country. And they didn't ask people if they were students. They were looking at people's-- more or less, are they young? Are they carrying a backpack? And that would be the labeling for your student.

So that's something that a sensor might not do so well. Although now with machine learning, who knows? But we haven't seen that so. So you can do qualitative observations when you're doing manual data collection. Manual data collection tends to be unreliable, especially when people aren't very well trained and when you have a group of different people collecting data. So each person might have different biases. It's hard to reproduce the exact bias across persons. With automatic data collection, you do the errors. And often, they are not corrected.

But if you do correct them, and you estimate those biases just for them, you can end up with a better result. Because of the small sample sizes in manual data collection, you tend to have to have limited spatial and temporal coverage of data. So for example, if you're interested in ridership in the system, it's unlikely that you will cover ridership in holidays for [INAUDIBLE] system because there are only a few holidays. And usually, you're not mostly interested in holidays. So chances are, you won't have data collection for holidays.

Whereas once you install automatic data collection systems, they keep collecting data. So you get data at midnight on President's Day. So they're always on. They're always collecting data. Manual data needs to be checked, cleaned, analyzed, coded, and sometimes put into systems before they can be analyzed. That could take a while. You need to hire people to do that. Whereas automatic data collection systems often send their data to databases in real-time or very close to real-time. [INAUDIBLE] you can start analyzing things the next day.

So you arrive in the morning to your desk at a transit agency, and you have performance metrics for yesterday. So you wouldn't be able to do that unless you have people working very hard if you're using manual data collection system. When we talk about automatic data collection systems, there are many. But there are three types that we refer to very, very often. And so the first one in AFC, Automatic Fare Collection Systems. This is your fare box or your fare gates in your smart card, your Charlie Card. You're in Boston. You tap to enter the bus. And you tap to enter the subway system.

Increasingly, it's based on contactless smart cards. And those contactless smart cards have some sort of RFID technology with a unique identifier. When you tap that card to the sensor, the sensor will read that identifier. And it'll do things like fare calculation for you. But that record gets sent to a database. And it's there for people like us to analyze and make good use of it for planning. So it tends to provide entry information almost always. In some systems, like the Washington, DC metro or the TFL subway, you tap in to enter and exit. So you have both origin and destinations.

And if you always have the systems on, then you have full spatial and temporal coverage of all of the use of the system at an individual passenger level. So very disaggregate-- sorry about that. Traditionally, these systems are not real-time. So it might take a while for those transactions to make it to the data warehouse, where they're available for planners to analyze it. The calculation of how much fare in some systems is in real-time. In other systems like the Charlie Card, the stored value that you have is stored on your card.

So it may take a while if you tap at a bus for that bus to go to a garage and get probed-- and for the data that has been stored in that bus to be extracted from that bus to the central server. There is a move-- and we'll talk more about this when we get to fare policy and technology-- towards using mobile phone payments and using contactless bank card payment systems. And those systems often do the full transaction over the air in real-time. So we're starting to look at the possibility of having all this data in real-time or almost in real-time. But it's not there yet.

AUDIENCE: [INAUDIBLE] can I ask a question about that?

GABRIEL SANCHEZ-MARTINEZ: Yeah, of course.

AUDIENCE: In terms of smart card, where this balance is stored on the card--

GABRIEL SANCHEZ-MARTINEZ: Yeah.

AUDIENCE: --if one can figure out how to hack that card--

GABRIEL SANCHEZ-MARTINEZ: Yeah.

AUDIENCE: --then what can [INAUDIBLE] fares through an elaborate technology that I couldn't do and most people couldn't do. But maybe some could.

GABRIEL SANCHEZ-MARTINEZ: Yeah, definitely. So the Charlie Card system is an example about-- actually, MIT students were the first to hack it.

AUDIENCE: I'm not surprised.

GABRIEL SANCHEZ-MARTINEZ: So it's older technology. It used a low-bit encryption key. That's a symmetric encryption key. And they just brute forced it. They figured what the key was. They happened to use the same key for every card. So once you broke that key, you could take any card. And with the right hardware, you could add however much value you want to that card. And--

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yeah, yeah, exactly. We don't think it's been a major problem.

AUDIENCE: But it happens.

GABRIEL SANCHEZ-MARTINEZ: I haven't seen MIT students selling special MIT cards. But that would be criminal, of course. Yeah, so newer systems have much stronger encryption. And they have different encryption keys for each card. And certainly, when we're moving towards contactless bank cards, we're talking about a much more secure encryption. It's your credit card that you're using to tap or your Android or Apple Pay.

AUDIENCE: Account based [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: Account based-- and essentially, what you have is a token with an ID. And then, the balance is not even stored on your card. The account server is handling the balance and those things. So much more difficult to break. Yup. OK, AVL systems, or Automatic Vehicle Location systems-- so these are systems that track vehicle movement. So for bus, they tend to be based on GPS. You have GPS on a bus, on the top of the bus, a little hub. And it collects data every five seconds or every 10 seconds.

And these positions might get sent either in real-time, or maybe they get stored on the onboard computer and then are extracted when the bus reaches the garage. So just GPS-- sophisticated AVL systems for bus also have gyroscopes to do inertial navigation and dead reckoning, especially when the GPS precision drops. And that happens especially with the urban canyon effect. If you have tall buildings, GPS signal bounces around. The dilution of precision messes up the position of the bus.

Or maybe you're entering a tunnel, and you want to continue to get updates of positions inside the tunnel. So this is a temporary system that kicks in and interpolates positions and figures out how the bus is moving. For a train, it's usually based on track circuits. So we're going to talk more about track circuits. But essentially, a track knows if a train is occupying that segment or not occupying that segment. And there are often some sensors that read with RFID technology the ID number of a car. And sometimes, you have a sensor in the front of each car and [AUDIO OUT] each car.

And so a computer will look up the sequence of readings and follow track circuits as they are being occupied and unoccupied-- and in that manner, track trains throughout the system. These systems were put in place mostly for safety to prevent train crashes. And because of that, you would need it to know buses or where a train was. They are available in real-time. They were designed from the beginning to track vehicles in real-time. So that's what we have. I guess what's newer is that now, we're collecting them and keeping them in a data warehouse so that we can analyze running times.

AUDIENCE: [INAUDIBLE] these systems have benefit to the consumer?

GABRIEL SANCHEZ-MARTINEZ: They do. And that's the newest thing that has happened-- that nobody thought about consumers when they were put in place. So yeah, we are talking about tracking, knowing how many minutes I have to wait for my bus, for example. And those things are pushed through a public API, so that if I'm a smartphone app developer, I can go ahead and pull data from this next bus app and make an app. And so people can download it, and they know how many minutes they have to wait. Yeah, so definitely. So we have seen a lot of AVL being pushed in that manner. We have not seen so much AFC data or APC data being pushed.

Obviously, you wouldn't want all the details of AFC being pushed. But you might want to know how crowded is my next bus, or how crowded is my next train. And you might actually alter your decision whether to wait for a crowded train or walk a longer time based on that information. So that's coming. I think, in the next few years, that's going to start happening. So passenger counting-- many different technologies exist. For bus, we tend to have these optical sensors in the back. You might see them if you pay attention-- broken beam sensors. They look like two little eyes with two little mirrors on each door.

And so when you cross the beams, if you press one beam first and then the other, that sensor will know-- is a person coming into the bus? Or is a person exiting the bus? And you have that at each door. And it counts those beams going in and going out. And often, this is slightly inaccurate. So you might get more boardings and lightings for a given trip. So at the end of a trip, whatever remains in terms of imbalance between boardings and lightings gets zeroed out. And the area is distributed throughout that trip that was just run.

And often, you still have to do some error correction after that. But it's a way of counting people getting on and off. And that's useful to get how many people are riding the system and also the passenger miles-- the passengers multiplied by distance, which is often a required reporting element in things like the NTB, the National Transit Database. So for rail systems, we have gates that count how many times they open and how many times they close. So you might have that kind of counting in rail.

You also have video-based counting-- so camera feeds that can be hooked up to a system that will essentially track nodes moving inside that frame. And you can count things that cross a certain line, for example. And you could do that to count flows. And then for train, we also have the weight systems. So this is only in trains. The braking systems in trains apply braking force in proportion to the load on each car. So if you have a very heavy car, you need to apply stronger braking force than in a car that is almost empty.

If you don't do that, then you apply a lot more force per weight on the lighter car. That car is going to be the one pushing the other cars or pulling the other cars through the coupling. And that will eventually break the [INAUDIBLE] at a faster rate. So what you want is, each car to slow down at the same rate by itself as much as possible. And for that, you need to brake in proportion to the weight. And therefore, you have these weight systems. They used to just do that.

And more recently, we hooked them up to a little storage device that keeps track of the weight and maybe Wi-Fi, so that each time it reaches a station or the terminal, it sends the data off. And we might have a rather somewhat [? unprecise ?] idea of how many people are in the car just based on an average weight of a person. And these are traditionally not available in real real-time. [INAUDIBLE] you have questions? Yeah?

AUDIENCE: You could also just reconcile it with the other system, right?

GABRIEL SANCHEZ-MARTINEZ: Of course, yeah.

AUDIENCE: So if you have--

[INTERPOSING VOICES]

GABRIEL SANCHEZ-MARTINEZ: Yeah.

AUDIENCE: --people early can transport to get on to.

GABRIEL SANCHEZ-MARTINEZ: Yeah.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yeah, definitely. Yeah. And that's cutting edge research that's happening right now. How do you do data fiction and merge different systems? They all have errors. And how do you detect when one is more erroneous than the other? And how do you mix these data sources to get the most precise, not just loads, but paths within a network and things like that. Yeah. So any questions on these three very important automatic data collection systems?

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yup.

AUDIENCE: So if there [INAUDIBLE] AVL, what kind of reason can be [INAUDIBLE]?

GABRIEL SANCHEZ-MARTINEZ: So the question is, why might some of these technologies produce errors? And in particular, you're asking about AVL. So each of these has a different behavior. And within each of these categories of technologies, each vendor's system might have specific things that happen. With AVL, the most common thing is end of root problems-- detecting when a trip actually begins and ends. So AVL systems, you have this GPS coming in every five seconds. Depending on your chip set, you might get it more frequently than that. But you also actually sometimes hook it to the doors.

So if the door is opening, you say, well, I must be at a stop. And therefore, let me find which one is closest. So there are ways to correct it. But when you get to the end of the route, it's not clear always-- have you finished your trip? Or rather, are you starting your trip already? So maybe if the terminal is at the same place on the trip-- the previous trip ends at the same place that the next trip begins, there might be a time where the doors open and close various times. And the trip isn't ready to leave yet. And so you really have to wait to see the bus leaving that terminal and moving.

Sometimes, there are false starts. So maybe another bus comes along, and it needs that space. So the driver moves the bus a few meters forward. And the system thinks my trip has started. And then, when you're looking at aggregate data, you're looking at, say, running times at the trip level. You see these outliers with very long times. And if you were to plot them by stop, you see that the link between the first stop and the second step is sometimes very high, 15 minutes.

And so you can throw those out. Or you can do some interpolation or imputation of data. Some systems that care very much about that will purposely place the terminal stops sufficiently far apart to prevent that from happening because it is a problem. And this data is crucial to planning service and figuring out how much resource you're going to put into each route. So yup.

AUDIENCE: For tap cards, [INAUDIBLE] and metros, some of them we have to tap out to exit. It is because of variable [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: Yes.

AUDIENCE: But in some systems, it's still a flat fare. You still have to tap out. Is the reason behind that mostly data collection? Or is there anything [INAUDIBLE] you're going to still have to tap out [INAUDIBLE]?

GABRIEL SANCHEZ-MARTINEZ: So yeah, no examples of it come to mind. You might know one.

AUDIENCE: MARTA?

GABRIEL SANCHEZ-MARTINEZ: OK, I haven't visited. So yeah, data collection might be a reason to do that. But I'll have to get back to you on why MARTA did that. But yeah, most systems that have controls in and out are for fare policy reasons and not for data collection reasons. We're starting to see more interest in data collection and in investing on these technologies just for data collection. So maybe-- but I'll have to check and get back to you.

AUDIENCE: You mentioned some systems separate their depots to not confuse the end [? from the start point. ?]

[INTERPOSING VOICES]

GABRIEL SANCHEZ-MARTINEZ: Their terminal stops, yeah.

AUDIENCE: What are some examples of those?

GABRIEL SANCHEZ-MARTINEZ: TFL will do that in London, yeah. Yeah, so they'll monitor this. And if they see that this is occurring often, they will separate the stops a bit. And the reason they do that is because they have people whose job it is to impute data when it's incorrect. So if they don't do that, and the system is consistently producing bad data, then that means they're going to have to spend human resources on correcting that data. So at some point, it's just easier to move the stop a little bit. It doesn't have to be a long distance.

AUDIENCE: Got it.

GABRIEL SANCHEZ-MARTINEZ: It does not make the same and make it far enough apart that the geo fences can be told apart from each other. Alright?

AUDIENCE: Really small scale data of the EZRide who I work for, actually you could see real-time bus loads [INAUDIBLE]--

GABRIEL SANCHEZ-MARTINEZ: Oh, interesting.

AUDIENCE: --which was actually helpful if you're dispatching, and you know a bus is getting through people on it. [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yeah, for real-time control.

[INTERPOSING VOICES]

AUDIENCE: But the terminal at our station had a drop-off point and a pick-up point. The drop-off point was before layover [INAUDIBLE] was after for this exact reason to make sure that it will go through the drop-off point, reset, until people get off of it.

GABRIEL SANCHEZ-MARTINEZ: Yeah. Yeah, so it happens.

[INTERPOSING VOICES]

AUDIENCE: Definitely. [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: That sounds about right. OK, if there are no more questions on the three very important categories of automated data collection systems, let's talk a little bit about the data collection program design process. So this comes from before automatic data collection. And nowadays, we think a little bit less about this. But it's still important. So if you do need to collect some data, there's a structure that you can follow to do it properly and to make sure that you collect data efficiently, so that you don't spend too much resources on data collection and that you can answer your policy or your planning questions.

So based on your needs and the properties of your agency, I say here, determine property characteristics. That's a North American term. A property is an agency. So if you see that, that's an agency. So based on the characteristics of the service you're running and your data needs, you can select some data collection technique. We'll get into what some of these are. Then, you can develop route-by-route sampling plans based on how variable the data is in each case.

And you can determine how many checkers do I need. A checker is a person who goes out and collects data. And then from that, the cost-- so human resources. It's a planning exercise. And what we do usually is that we conduct a baseline phase. So that's the first time you go out and collect data. You don't know much about what you're wanting to collect data on. So it might be only matrices, or loads, the people getting on and off. So you have to go out and do a bigger effort. And that's called the baseline phase effort. Once you've done that and you've established some tendencies, you might want to monitor that to see if it changes.

So then, you do a lighter weight data collection effort, where you go out and less frequently, using fewer resources, you collect sometimes the same thing. Or sometimes, you observe something else that is related or can be correlated with what you really want. And then based on a relationship between the two, you can estimate what you really want. So you can monitor what you collected. And then, if you detect that there's been a trend or a change, and you need to investigate it further, you might go ahead and repeat the baseline phase to increase your accuracy.

So one of the catches of this is that to determine sampling plans, to determine required sample sizes to achieve some confidence interval, you need to know how variable your data is. And if you haven't collected it yet, you don't know. So you might have some default values that you resort to. And we'll get to that later in this lecture. But you might also do a pre-test, where you send some people out, and you collect some data to really start to get a sense of how variable is it, and how big will my sample requirements be, and how much will it cost for me to do this. So this is the process that you might follow.

And there are different data needs by the question that you're trying to answer. So one way of looking at that is, are you collecting things that are for specific routes, or for specific route segments, or at the stop level? Or are you using more aggregate system level data collection? Are your questions more system level? So system-level things are more about reporting, and they might be tied to things like federal funding. Whereas route-level things and stop-level things are more important for planning.

So when we talk about route and route segment level, we're looking at things like loads at the peak load points or at some other key points. How many people are in the bus? The running time is by the segment to do schedule that has time points or maybe end-to-end to your operations plan. Schedule adherence-- are these buses running on time? Or are my schedules not realistic? Total boardings or revenue, two things that are highly correlated-- so number of passenger trips.

Boardings by fare category-- so you might say, well, I want boardings, but I want to know how many seniors are using this, and how many students are using this, and how many people are using monthly passes, and how many people are using pay-per-ride. So you have different fare categories. And you might want to segregate the data by that. You might want passenger boarding and lighting by stop. So that's what APC would give you if you have an automated system. But you might also use a write checker, who sits on the bus and counts people boarding in a lighting.

Transfer rates between routes-- to see you maybe you're looking at changing service so that people don't have to transfer. Passenger characteristics and attitudes-- this usually requires some degree of survey, where you ask people things, passenger travel patterns. At the system level, we have things like unlinked passenger trips, passenger miles, linked passenger trips. This had the whole system level. So sometimes, you do route level or route segment level analysis, and then, you aggregate to get the system-level things. That's usually how you proceed.

But the requirements in terms of how many of these you have to sample might be different. So if you want to achieve a certain accuracy at the system level, you don't need to achieve the accuracy for each of the routes that are in that system because you might have-- so if you want to say 90% confidence in some system-level data element, you might only need 80% or 70% of the element level. And once you bring those altogether, you achieve the 90% that you need.

So data inference, I talked about how sometimes we can infer items if we don't observe them directly. So from AFC with AFC is a low-fare collection system, we have boardings because people are tapping into the bus or tapping into the subway system. And if we have APC, we count people getting on. So we can look at total number of boardings that way, if that makes sense.

That's pretty direct. Sometimes, you want to correct for errors in the APC system, or you might have things like variation affecting that number-- like it goes from AFC to how many people were actually in that bus. How many people actually boarded? So you might do a little bit of manual surveys to check what that relationship is and apply some correction.

For passenger miles, we need to know how many people are at the bus between each stop here. So AFC gives you boardings and only boardings. APC gives you ons and offs. If every bus had APC, then you could calculate passenger miles directly. But often, you have systems where only a portion of the fleet has APC. So maybe 15% of your fleet is equipped with APC. And from that, you get the sample OD matrix. And you can use that OD matrix to convert from boardings only to the distribution and the ons and offs at all bus routes. And from that, you can get passenger miles.

Or you might just use your buses that have APC, if that suffices for your data collection unit. Same thing with peak point load-- similar idea. The AFC only measures boardings. So it doesn't give you the peak point load automatically. But from APC, you could get it. And it you can establish a relationship between boardings and the peak load point, then you can use that model to infer the peak load point from just boardings. So this is a key thing to be efficient about data collection. Any questions on this idea? Yup.

AUDIENCE: So to get passenger miles, you're also going to have a GPS system as well to know the distance? Or are we just basically [INAUDIBLE] this is the routing [INAUDIBLE]?

GABRIEL SANCHEZ-MARTINEZ: Both.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yeah, both.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: What tends to happen is that the APC, it'll come in. And it'll say, at this stop, this many people boarded. This many people are lighted. So you have other layers in your database that say where the buses and what the distance is between stops and the stop pair level. So you then essentially know how many people are riding on each link and how long that link is, and you multiply the two. So yeah, passenger miles. Yeah, more questions.

AUDIENCE: Yeah, for these checks that are going on like the more manual checks--

GABRIEL SANCHEZ-MARTINEZ: Yeah.

AUDIENCE: --I know often, there's derivation checkers who are coming into a check.

GABRIEL SANCHEZ-MARTINEZ: That's right, yeah.

AUDIENCE: Do they also use that data to cross-reference the passenger counts? As in, [? this ?] person gets on, and they check everyone's voice to [INAUDIBLE] DFL.

GABRIEL SANCHEZ-MARTINEZ: Yeah.

AUDIENCE: They then know exactly how they go on the bus.

GABRIEL SANCHEZ-MARTINEZ: Yes. Yeah.

AUDIENCE: Do they use that data?

GABRIEL SANCHEZ-MARTINEZ: Yeah, they can. In the APC, sometimes there's reliability problems, especially when vehicles are very full because sometimes, people will block the sensor by the door. Actually, people like to stand by the door all the time, even when the bus isn't full. And that kind of affects APC. You might notice this on the one. If you take the one-- so yeah, you sometimes have a little bit of a manual effort to figure out. Just learn about your APC system, and what are the errors, and when do you see them. It often happens that you have more variation when you have very high loads. And that's when APC is least accurate. So it all comes together. Yeah. Questions on the back? I think I saw a question. No?

AUDIENCE: Yeah, I noticed that in Chicago, when the bus would be crowded, then people get off the bus. They let people off--

GABRIEL SANCHEZ-MARTINEZ: That's right.

AUDIENCE: --and then back on.

GABRIEL SANCHEZ-MARTINEZ: Yeah. Yeah. These double things. But somebody might be by the door just blocking the two little sensors--

[INTERPOSING VOICES]

GABRIEL SANCHEZ-MARTINEZ: --the two little eyes. And that's it, no records of people getting on or off. So if you're doing a little data collection, as I said, we use checkers. And actually, your second assignment, you will be checkers of some kind. The typical checkers which you won't be in this assignment are ride checkers and point checkers. So a ride checker sits in the vehicle and rides with the vehicle. And the typical thing that these ride checkers are looking at is, how long did it take to cover some distance? So what was the running time for that trip? And also, people getting on and off-- so they act as APC essentially. And they act as AVL.

So AVL and APC together might replace most of the functionality of a ride checker. Although a ride checker often can conduct an onboard survey, asking passengers about where are they going, or their trip purpose, or things related to social demographics, which are qualitative and cannot be collected with the sensors. Point checkers stand outside of the vehicle. They stay at a specific place, and they can look at headways between buses-- so how long did it take between each bus to come by, and how loaded were these buses?

So if you're interested in the peak load point, and you know where the peak load point is, and you just want to observe, measure what are the loads of the peak load point, then you can just station a point checker at the peak load point. And if that person is strained, we'll be able to more or less say how many people are in the vehicle from looking at the vehicle.

With automated data collection systems-- yeah, with a fair system, we have passenger accounts. We have transaction data, which is very rich. It will tell you not only that somebody is entering or exiting, but also how much they're paying, sometimes information about the fare product type, which might help you infer if this person is a senior, or a student, or a frequent user, an infrequent user-- so many things that are very useful for planning. And we'll get to play with some of these later in the course. And then, there's Automatic Passenger Counters, APC.

So as more and motor systems switch to automatic data collection, we still use some manual data collection, but not in the traditional sense. Now, we reserve those resources for things like surveys about social demographics and other things. And we also carry out web-based surveys, which would have some biases. But if people registered their cards, and you have email accounts, you can maybe send a mass email to everyone and carry out surveys. The MBTA does that. Maybe some of you are in the panel of people who are e-mailed every now and then. Is anybody in that panel? No hands. I'm in that panel.

But I know somebody must be. So yeah, they send an email, and they ask about your last ride. And they say, where did you start from? What were you doing this trip for? How long did you have to walk? Are you happy with the system? Was your bus on time? Yeah, things like that-- how satisfied are you? It's a survey with qualitative questions that you couldn't collect automatically. It's [INAUDIBLE] seeing things about your experience outside of the bus, which there are no sensors for.

All right, sampling strategies-- a bunch of different ones and the simplest one is called simple random sampling-- very, very simple. So when you have sample random sampling, what happens is that every trip, if you're looking at surveying trips, for things like how many people boarded this trip-- let's take that as an example. Then, if you're using simple random sampling, every trip has equal likelihood of being picked and being surveyed. So if you go through your process, and you determine that you need to observe 100 trips to get an average reliably. And you're going to use that to plan something, then you need to look at 100 trips.

So if you use simple random sampling, you take your schedule, and you randomly pick 100 trips. And that's your sample. Those are the ones that you send people out to collect data. Now, there's a little bit of a problem with that. It's not the most efficient method because if you're going to send someone out, and that person is going to be active, and require some time to get to the site and some time to return, then once they're out there, you want them to collect as much as they can. So that's not simple random sampling. That's cluster sampling.

Before we get to that systematic sampling-- so typically, instead of picking randomly, we say, OK, we need to get 10% of the trips. So let's just make it such that we count. And maybe it's every five trips, we have to survey it. So now, it's evenly spaced. And this is useful for some things. One example is weekday, picking the weekday that you're going to survey on. So the technique that is often used is sample every six days. Why would that be? Yeah. So if you do it every seven, then you always have a Monday. And that's going to get some bias if Mondays happen to be low ridership days or high ridership days.

So if do every sixth day over a year, you have a good sample of every week day. So that's an example of systematic sampling. But you still have that issue of it might not be the most efficient. Cluster sampling, sometimes it's more efficient once you send out a person to collect data to do as much as possible. And you survey a cluster. So one example is, if you're distributing surveys to passengers, and you need to distribute 100 surveys. If you do 100 simple random sample, then those people might be in different parts of the system. And one might be the first person you see getting off at South Station.

And then another one by me might be the first person you see getting off at the Kendall station. So that's very inefficient. So a cluster might be everybody on board a bus, and that will get a bunch of people together. However, it's not as efficient statistically to do that. So you can't just add up to 100, and you're done because there might be some correlation within the people riding that vehicle that they will tend to answer in a similar way. So you might need to increase your sample size when you use this technique. But still, you might have a more efficient sampling plan.

Then, there is the ratio estimation and conversion factors. We gave examples of this already. This is in the context of baseline phase and then monitoring phase. So you start out with a baseline phase. And in the baseline phase, you collect the thing you really want and something that is very easily collected with lower resources. And you make a model of the thing you really want as a function of the thing that is cheap and easy to collect. And then, on the monitoring phase, you only measure the thing that is cheap, and easy, and quick. And you then use the model to estimate what you really want.

So converting AFC boarding to passenger miles, we give an example of that. We're converting loads at checkpoints to load somewhere else. So maybe only measure loads with a point checker at the peak load point. And you have some relationship to convert those loads to loads at other key transfer stations as an example. And then, the stratified sampling-- so one of the things that determines how big of a sample you need is the variability in the data that you're collecting.

So correlation, when you're looking at a whole system with multiple routes or multiple segments-- maybe when you look at one route, there's some variability of running times. But they have a central tendency as well. And when you've got a second route, you have also some variability and a different central tendency. So you bunch all the data together, some of the variability across data points in our data set are going to be the inherent variability of each route. And some of it will be systematic-- the differences between both routes.

So if you do a simple random sample, and you don't separate the systematic variability from the inherent variability, then you're going to get a wider variability. And you will require a bigger sample size. Stratified sampling is an approach where you determine sample sizes for each of these separately. And it's more efficient if you do it well because you eliminate the need, or you at least reduce the need, to collect data for the sake of the systematic differences between different parts of the system. Any questions on these methods? Yes.

AUDIENCE: [INAUDIBLE]

 

GABRIEL SANCHEZ-MARTINEZ: Yeah, so let's maybe pick another example. Let's say that you're looking at the proportion of passengers in a bus who are students. And you're distributing a survey. And they tell you whether they're students or not. And you want this for the whole system or for at least a group of routes. And it tends to be that some routes don't serve universities and don't serve schools. So they have a lower proportion of people. And then, some routes that do go through universities, and they have a higher proportion of students.

So if you just want the system-wide proportion of people who are students, and you join all these data points together, there's going to be a lot of variability in what proportion that is across every trip that you survey, correct? So in some sense, it will indicate that because of that variability, you're going to need a higher sampling size. You're going to have to survey more trips to get at your desired accuracy level and tolerance. But now, if you say no, I'm going to split routes in two, into two stratas. One is the routes that serve the universities. And these tend to have around 50% proportion.

And then, there's the routes that don't serve universities. And these tend to have proportions near 0. So if you're in your 0, you might require a lower sample size to cover those. And you can just very efficiently cover most of your bus routes that way. And then, focus your efforts on just the ones that have higher proportion. And you achieved your system-level tolerance requirements with much fewer, with by far fewer resources required to collect the data. Does that answer your question? Yeah.

AUDIENCE: [INAUDIBLE]

 

GABRIEL SANCHEZ-MARTINEZ: So what he meant by inherent is that within each bus route or within each strata, there will be some variability. Even within the trips that are serving universities, every trip might have a different proportion. So there's going to be a little bit of variability in that. But if you mix that with trips that are not serving students, then you pull all that data together. Then, it's going to look like the variance of that data set is much higher. All right, so we've tossed these terms around-- tolerance, confidence, level accuracy. So let's define them more precisely.

Accuracy-- when we talk about accuracy, that has two dimensions. So somebody might say, the average boardings per trip is 33.1. And then, the question that follows is, do you mean exactly 33.1? How certain are you of that? And how accurate is that? So when we talk about tolerance, there's relative tolerance, and there's absolute tolerance. Relative tolerance is expressed in terms of a percent of the amount you were collecting or a fraction. So you might say mean boardings per trip is 33.1, plus or minus 10%. And that's the 10% of 33.1. That's why it's relative tolerance.

Then, there's absolute tolerance. So mean boarding per trip is 33.1, plus or minus 3.3. Now, in this case, these two are equivalent. 3.3 in absolute terms is 10% of 33.1. But this was expressed in absolute terms, and the previous one was expressed in relative terms. So don't always assume that if you see a percent, it's relative because if what you're measuring is in itself a percent, unless you're using a percent of a percent, then it's absolute. So here's an example. Mean percentage of students is 23%, plus or minus 5%. That's absolute because it's 5%, not 5% of 23%.

First, we talked about, is that exactly 33.1? Or is it something different from 33.1? Then, the second question is, how sure are you, how confident are you that the number you give, plus or minus the tolerance you give, is the right answer? So now, you say I'm 95% confident that the mean boardings per trip is 33.1, plus or minus 10%. So now, you combine the tolerance with the confidence level. And that's the full expression of your accuracy. And that's what you need when we look at the data collection.

So you have two different things that you could play with. And what happens typically is that you choose a high confidence level-- 90%, 95 percent are typical. And then, you hold that fixed. And you calculate what level of accuracy you need. Or rather, you decide what level of accuracy you need, depending on the question you want to answer, and the impact it could have on the system. So if you're looking to [INAUDIBLE] something that will have very significant effects on the service plan or maybe on investment in the system, then you might need a higher accuracy.

But if you're collecting data just for reporting, maybe it doesn't matter as much. And you don't need to spend as much money on data collection. So as an example here, the National Transit Database-- NTD, we call it NTD-- for annual boardings and passenger miles, it says, you should collect data to achieve an accuracy of 10%, relative tolerance at 95% confidence level. You need both. So take home message about this.

The other thing, the t distribution-- so this is a probability distribution that is bell-shaped. It kind of looks like the normal distribution. And it approaches the normal distribution as the sample size gets very large. This is the distribution that arises naturally when you're estimating the mean of a population that is normally distributed with unknown mean and variance and some known sample size. So to the right here, we have your equations that I'm sure you've seen before for sample mean, sample variance.

And I guess, what's important to think about is that the distribution of what you're collecting-- for example, you might be collecting data on a number of people boarding route 1. So that might have some distribution. As you collect more and more data, so as you survey more and more trips, the distribution of how many people board each trip does not necessarily have to be normal.

But it turns out from the Central Limit Theorem and other laws and properties of statistics and probability that the distribution of the estimator-- so the distribution of the mean that you calculate based on that sample that you collected-- is normally distributed as the sample size increases. So if you have a lower sample size, instead of using the normal distribution, use t distribution. Sometimes, we call that a student, the t student distribution. And this distribution gets wider as the variability increases and as the sample size gets smaller. It has a property called degrees of freedom, which is sample size minus 1.

And you can see from this chart right here when you have degrees of freedom equals 1, which means you collected two data points, it's wider than when V approaches infinity. And what you have in black here, the thinnest and least variable of these, is essentially a normal distribution. And this is the distribution not of what you collected. It's not the distribution of the number of people who boarded route 1. It's the distribution of the mean that you estimate.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Exactly, it's a sampling distribution of the mean. And if you were to repeat that experiment with the same number of trips but different number of trips, you might get a slightly different mean. So if you were to repeat that many, many times, the distribution of those means would be shaped in this manner.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yeah, well, student t distributed. And as sample size increases to infinity, normally distributed. Harry.

AUDIENCE: So just for V equals 5, I think you [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: 4.

AUDIENCE: 4.

GABRIEL SANCHEZ-MARTINEZ: Sorry, 6. 6.

AUDIENCE: Approximately 5 [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: Yes, 6. Yeah. I mispoke. [INAUDIBLE]

AUDIENCE: When there's a sample variance, sigma x squared equals roughly. Is that not supposed to be an equals? Is that not the way the sample variances define? Because I thought it's the--

GABRIEL SANCHEZ-MARTINEZ: So-- --it's below the variance of distribution. But that's roughly [INAUDIBLE].

AUDIENCE: Yeah, I guess the issue is that you don't know the true mean. So you're using an estimate to calculate the sample variance. And therefore, it's almost, almost the sample variance.

GABRIEL SANCHEZ-MARTINEZ: Right. But I thought--

AUDIENCE: You're using an estimator to do the-- that's what you have to do.

[INTERPOSING VOICES]

AUDIENCE: He's incorporating the fact we're dividing by n minus 1 rather dividing by [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: No, so n minus 1, that has to do with the degrees of freedom issue. And that's to go from population variance to sample variance. But the other thing that happens is that if you're doing the population, then you know exactly what your mean is. It's exact, right?

AUDIENCE: Yeah.

GABRIEL SANCHEZ-MARTINEZ: And then in that case, you would know what the exact variances is as well. Yeah. So the n minus 1 is just to remove a bias that would arise from collecting only a sample.

AUDIENCE: But here for example, you can say this is equals to [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: Yeah, yeah, yeah, yeah.

AUDIENCE: You're working with the sample to know it would be an approximate [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: Yeah, in practice equal 2.

AUDIENCE: As your sample distribution increases, then obviously, your sample increases--

[INTERPOSING VOICES]

GABRIEL SANCHEZ-MARTINEZ: And therefore, this becomes more and more accurate.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Exactly.

AUDIENCE: It should be approaching more [INAUDIBLE].

GABRIEL SANCHEZ-MARTINEZ: Yeah, so I guess what's important to realize is that this is an estimate of the population variance, which in itself uses another estimate. And I guess, that's why that's there. But it's a very small detail. I didn't mean to distract you.

AUDIENCE: So for the n, is it the sum of all the different samples of [INAUDIBLE] or is it just--

[INTERPOSING VOICES]

GABRIEL SANCHEZ-MARTINEZ: So you don't ever repeat the experiment like this. This is more of a theoretical explanation to why there is a distribution to the mean, even though you only have one. You only have one mean, right? Because you're going to collect data. And once you finish collecting data, you're going to calculate the mean of all that data. So you only have one mean. If you were hypothetically to repeat that experiment, and you calculated separate means for each one, then you would get a distribution that would look like this. In practice, you would just increase your sample size and still compute one mean, which would be more accurate. Yeah.

OK, let's move on. So tolerance and confidence level-- so we have these distributions. These are the distributions of the statistics, of the mean in this case. They are bell-shaped. As your sample size increases, the degrees of freedom goes up. And your accuracy goes up. And the variance of that statistic distribution decreases. So it gets thinner. So here in red, you have a distribution with a smaller sample, and therefore, less accuracy or less confidence would look like. And then as you increase your sample size, you see that it becomes more peaky.

So when we talk about tolerance, and let's come back to the concept of absolute tolerance in particular, we're talking about the distance between the center of that distribution, which is a symmetrical distribution, and some limit. So we're saying, if you have a tolerance of plus/minus 10. Then, you're going to measure 10, say 10 boardings, from the center to the right and from the center to the left. And that's your absolute tolerance. So when you calculate absolute tolerance, you can express that tolerance as a function of the variance and/or the standard deviation, rather of your mean.

So instead of saying 10, you could say 2 times the standard deviation of that distribution using the equation that we just calculated. And that's very convenient. Why would we do that? Why would I want to complicate things that way?

 

AUDIENCE: [? Outside ?] [? of ?] a cumulative

GABRIEL SANCHEZ-MARTINEZ: No, I mean, there's a mathematical convenience here. What is this a function of? It's a function of the standard deviation of the thing you were collecting and your sample size, right? And what do we want to do? We want to determine how many things we need to collect, right? So here we go-- we have n. And now we can solve for n, we have the sample size that we require for a given tolerance. So we're going to decide what the tolerance is and calculate sample size, a minimum required sample size. You can always collect more data.

All right. So again, to review, this is the same equation I had in the last slide. You have absolutely tolerance. You can express that as a multiplier times the standard deviation of the mean. And then you solve for n, and you get this equation right here. t is your tolerance and you can-- oh, sorry. t is the number of standard deviations from the mean. d is your tolerance, which you choose. And this is something that you know, or collect, or approximate.

So these are all given. Where does t come from? Well, we said that we're going to use the t distribution, right? So the t distribution has a table-- or it has a certain shape, rather. And using Excel or looking up at some table, you can figure out what t is for two times the standard deviation from the center.

So you can just plug it in from Excel or from-- it's a property of the distribution, essentially. Once you pick a confidence interval, you know t. If you want to go to 95, it's a certain value. If you want to go to 90, it's a different value.

OK. When we look at relative tolerance, relative tolerance is just absolute tolerance divided by the mean that you are collecting, correct? Because instead of saying plus or minus 10 boardings, we're saying plus or minus 5% of the mean. So we just take absolute tolerance and divide by x bar, the sampling mean, the sample mean. And we solve for n again.

So what we have now, it looks very similar as to the question right here. But now we have the mean and the denominator. OK, this quantity, standard deviation divided by mean, sample standard deviation divided by sampling mean, is called the coefficient of variation. And there's a convenience to this. And there's actually a reason why sometimes relative tolerance is preferred to absolute tolerance. It's because of this, because there's a mathematically convenient characteristic of property coming out of this-- that you don't need to know the standard deviation of what you're collecting to figure out your sample size.

We're kind of running in circles here, right? We're saying that to determine sample size, you need to know the standard deviation. Well, I haven't collected data. So I don't know how variable the data is. So that's an issue. Now I have to estimate what that is.

It tends to happen that the coefficient of variation is a more stable property than the variation in itself, than the variance or the standard deviation itself. So you're more likely to get away with using default values for the coefficient of variation than you are with assuming a specific standard deviation.

AUDIENCE: It should be noted that it's unitless, coefficient of variation.

GABRIEL SANCHEZ-MARTINEZ: Yes, it is unitless. Thank you. OK. So what happens is that relative tolerances are typically used for averages. So here's an example-- you measured 5720 boardings plus minus 5%.

So if you were to get the absolute equivalent of the absolute tolerance of that. That would be 5% of 5720. That would be 286 passengers. That's a weird thing to report. 5% is more understandable, right? And it kind of makes more sense. So that's what we want naturally, anyway. So as I said, the coefficient variation is typically easier to guess than the mean and the variance separately. So we use that.

Here's an example using the t distribution, where the sample is not large enough to assume a normal distribution. So we say, let's have a relative tolerance of plus minus 5%, a confidence level of 95%, and a coefficient of variation of 0.3. So we start out assuming large sample, and therefore degrees of freedom is infinity. We can use the normal distribution.

If we look at the normal distribution, with plus minus 5%, confidence level 95%, the t is 1.96. So we look that up on a table, or we use Excel norm dist, or-- yeah. t dist for t and norm dist for normal. We got 1.96.

We plug in the relative tolerance, the 0.3-- we get 140. 140 is not quite infinity, right? So if we look at 140 as a sample size, that would imply that all the degrees of freedom is 139. Now we go back and look at the t dist, and we change 1.96 to the value from the t distribution for that degree of freedoms. And we get 140.73.

So you're sort of seeing that you were almost right. 140 is very large. In practice, you would just round up a little bit and get a nice round number, and you would even play with this once you're looking at planning who you're going to send out and how many hours you're going to collect. You want to get at least 141, but if you're going to have people in units of eight hours, for example, or units of four hours, then you might as well finish the batch for four hours, the last one. Maybe you'll get 150, 160 from that.

Here's an example of that equation with different assumptions of confidence and tolerance. And so we're using 90% confidence, and we're assuming a certain sample size here. So you can see that, as the tolerance decreases, which means that you require a greater accuracy for different coefficients of variation, the sample size can get really large. So if your data is not very variable, then you can sample just a few trips. And you know because they don't vary that much what the mean is. But if there's a lot of variability across strips, then you need more. So that's what you see as you go down the rows on this table.

Here we have tolerance. If you only have to be 50% accurate, plus minus 50%, then you don't have to collect that much data. If you want to be more precise, and you want to say plus minus 5%, then you need a bigger sample size, right? OK.

Proportions-- and the homework, actually, is based on proportions, so this is important. Consider something, a group of passengers, to estimate the proportion of passengers who are students. So from probability, when you are looking at an event that can either be 0 or 1, or black or white-- in this case, students or non-students-- there's a certain probability that that person is a student, right? And what you want to estimate is that probability or, in other words, what percent of the things you observe are students.

So from the properties of the Bernoulli distribution, the variance is p times 1 minus p. So if everybody is a student, or nobody is a student, either way there's no variability, right? So you would have 1 times 1 minus 1, 1 times 0, 0-- no variability. Though at the peak variability, the highest variance of this distribution, is when 50% of your people are students, so 0.5 times 1 minus 0.5, 0.25. That's the highest variance, OK?

So the tolerance is typically specified in absolute terms when you're estimating proportions, because the proportion is in itself a percent. So you use absolute tolerance. And you just substitute, essentially, this variance. You put in the variance of the Bernoulli distribution, which is p times 1 minus p. And that's how you get the sampling equation, sample size requirement equation.

Here's a problem. We don't know in advance what the proportion will be, right? And we need that to know how many people we need to survey to figure out-- or how many trips we need to survey to figure out-- sorry, how many students we need-- how many riders we need to survey to figure out what the average number of students are. OK, so--

AUDIENCE: And it's also a [INAUDIBLE] p times 1 minus p [INAUDIBLE] is a constrained number.

GABRIEL SANCHEZ-MARTINEZ: It is a constrained number, and that's exactly where we're going. So we use something called absolute equivalent tolerance instead of absolute tolerance. We assume that p is 0.5-- that's the maximum it could be. So let's go ahead with a worst case scenario.

And then what happens with p itself? Well, if your percent is high, then you can tolerate a bigger number, right? So if it's 32%, you're probably OK with plus minus 5%. If your average were 1.2, plus minus 5% is not that good, right? You need a higher-- you need a much stricter, tighter confidence interval for that. So probably not good to do plus minus 5% in that case.

AUDIENCE: [? Well, do ?] [? you mean ?] you have a plus minus 5% absolutely percentage?

GABRIEL SANCHEZ-MARTINEZ: Absolute, yeah.

AUDIENCE: And you'd be going negative [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Negative, which is possible but difficult to interpret.

AUDIENCE: Sorry, so this isn't actually 32% plus or minus 5% of 32 [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: It is not-- yeah, it's absolute tolerance, not relative tolerance, right. So what's convenient about this is that these two factors work in opposite directions. So as you get bigger, as the proportion gets closer to 50%, the variance increases. So oh, well, we need a bigger sample. But your tolerance increases as well, so you don't need as big of a sample.

And so it's convenient. And the practical solution is assume p is 0.5 and work in terms of absolute equivalent tolerance. So you pick a tolerance under the assumption that our proportion is 50%.

And here's what happens. Yeah, if the expected proportion is 50%, and you say plus minus 5 percent, what you would get is this 5%, if it turns out that p is 5%. But if it worked more to the extremes, like 5% or 95%, what you would actually achieve from having planned the survey, assuming 50%, is 2.2-- so much better, much more acceptable to say 5% plus minus 2.2%, right? So it works out.

And there's a convenient equation if you assume a very large sample, or large enough sample, and you pick 95%, 0.25, which is the variance times the normal distribution t squared is 0.96, which is almost 1. So then you get this equation. You take 1, you divide it by the tolerance that you want, your equivalent tolerance, and that's your sample size. So it doesn't depend on anything about the data in itself. You just say if I want, on whatever I'm collecting, whatever proportion I'm collecting, a 5% absolute equivalent tolerance, then I need 400 surveys to be answered. Yeah?

AUDIENCE: So this assumes a random--

GABRIEL SANCHEZ-MARTINEZ: Simple random sample.

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yes, a simple random sample. So you would increase these numbers if you are using cluster sampling to account for correlation. You would have to increase them if you're giving people a survey, and not all of them answer the survey, because you need 400 surveys answered. So if only half of the people answer the survey, then you need to distribute 800 surveys.

AUDIENCE: Do you recommend calculating also that the standard error after this so that [INAUDIBLE] make sure?

GABRIEL SANCHEZ-MARTINEZ: Absolutely, yeah. You want to go back and check with the standard error and when your confidence interval is and see if you meet it or if you need to add a few days of data collection.

AUDIENCE: Right.

GABRIEL SANCHEZ-MARTINEZ: Yeah. OK, so with proportions, you need a very large sample size to estimate a proportion if you want accuracy. If you say absolutely equivalent intolerance of 4%, then you need 600. That's a big number, so it just gives you an idea of that. If you get greedy with the tolerance, you have to pay for the surveyors to go out. OK.

So the process is you determine the needed sample size just with the discussion of the equations that we discussed. Then you multiply the sample sizes. If you're using stratified sampling or if you have questions that have multiple variables, you need to then make sure that you achieve that sample size for each combination of things that you're measuring.

So if you're, for example, looking at not just boardings, but proportion of passengers that are car-owning, who are pleased. So you could just independently measure pleased, independently measure passengers who own a car. And you might have the tolerance you need on each one, but if you want the combination of that, now you need a higher sample, because you need that number for the combination of those things.

Then there's a clustering effect, so a typical thing if you're doing the clustering of a whole vehicle of passengers is to multiply by 4. And then for things like OD matrices, the rule of thumb is 20 times the number of cells. What does that mean? That if your OD matrix is quite aggregate, and it's at the segment level-- so say you divide a root into two segments, then your OD matrix has four cells. Four cells times 20, that's how many people you have to survey.

If you do error at the stop level, then you have many more stops and, therefore, many more cells and, therefore, a much higher sample size. If you have a response rate that is not 100%, which is always the case, then you have to expand by 1 minus that in the reciprocal-- sorry, 1 over that in the reciprocal.

And then you get a very large number, and you say I don't have the budget for that. And you have to make tradeoffs and figure out what you can do. And maybe you have to-- maybe you can't collect this combination and know that accurately, right? So you revise your expectations.

OK, with response rates, you are concerned with getting the correct answers. You also want to be getting a high response rate. If you don't get a high response rate, there might be a bias. So you have to worry about that.

If you have low response rates, that means you need to distribute more surveys, and that costs money. And there's the bias that I just mentioned, so people who don't respond may not be responding for a reason. And then done that might bias your results. And that might make you decide something in planning that is not the right decision based on what actually happens.

So we call that the non-response bias. OK, so what happens? People who don't respond might be different or might have responded differently to the question had they responded. So here's some examples. If you're surveying people who are standing, they are less comfortable. And maybe it's a crowded bus-- they are less comfortable. Or maybe they're getting off one of those stops that is coming up, so they are less likely to have the time to respond to your survey.

People with low literacy, teenagers, people who don't speak the language, are less likely to respond. And they might have different travel patterns. So if you understand those things, and you get lower samples for them, you might be able to do some sort of correction to those biases. But you have to pay attention.

How do you improve your response rate? Well you can make your questions shorter. You can do a quick oral survey. That's what we're going to do for this homework. You can try to get information from automatic sources whenever possible. So if you have an AFC system, let's not collect boardings, because we know that. And then of course some training, and just being kind, and having supervision helps a lot.

OK, here's some suggested tolerances for different things. So we're looking here at boardings or the peak load. And you see here that the suggested tolerance is 30%, plus minus 30%, when you have a route with one to three buses. And then as you have more and more buses, the tolerance decreases. That means you require a larger sample.

Why is that? Why do you need a bigger sample if you have a route with more buses?

AUDIENCE: You're less likely to sample a different bus.

GABRIEL SANCHEZ-MARTINEZ: Yes, and when you have higher-- when you have more buses, you tend to have higher frequency. There's bunching.

OK, so if you then survey loads, for example, and you only get a few because of the bunching effect and because there are more buses, and you're observing a smaller percentage of them for a given time period, say, you're less likely to have observed the bus that was really crowded, right? So that means that you need to decrease your tolerance. And therefore, it's more expensive to survey that. OK, good.

Trip time-- 10% for routes with less than 20 minutes, 5% with routes of greater than 20 minutes. Similar concept if you have greater than 20 minutes-- there can be just more variability, and you really want to get that right. When you have less than 20 minutes, your decision on cycle times and things like this are not going to have as much impact on the fleet size that you require. As you get bigger running times, a small percentage change in the mean could influence how many buses you need to dedicate to that and the cost of running that service.

On-time performance-- 10% absolute equivalent tolerance. These are typical values-- don't take them as gospel, please. And these are for reporting, not for anything that's very critical for operations. Some of them are. Yeah, 30% at least, I would say, is for reporting. I wouldn't make any critical decisions with 30%.

On-time performance-- we're talking here about whether a trip is on time or not on time-- so Bernoulli trials, right? And there's a proportion of trips that are on time, and what we do is that, we essentially say plus-- if we say plus minus 10%, then we're saying that the sample size should be 1 over 0.1. Yeah.

All right, default coefficient-- these are default values for coefficient of variation of key data items. Ideally, you have your own data that you look at, and you don't resort to this. But if you ever find yourself in a situation where you need to start out with something. Here are some based on studies that previous [AUDIO OUT] They took different routes and looked at loads and running times for different time periods and found what the coefficients of variations were. And here they are on a table for you to use.

In the interest of time, since I want to discuss the homework, I'm going to stop here with slide 25. And I'm going to not cover the whole process, which includes the monitoring phase. And in this slide here, we have how you establish conversion factor. The conversion factor in itself has a variance. So there's some uncertainty about the relationship that you estimate between your baseline data item and your auxiliary data item. So you need to consider that in your sample size. And here are some tables with some examples of what happens when you require different-- well, when you're variability of or your coefficient of variation of your relationship increases or decreases.

OK, let's look at the homework. I really want to use these last five minutes for that. So please take one and pass. OK, so the MBTA, there's a proposal here in Boston of taking Route 70 and 70A-- they run through Waltham, and they go into around Central Square. And some people are saying those two routes should be extended to Kendall Square, because a lot of people are actually going to MIT, or Kendall Square, or the Kendall Square area-- not just Kendall Square Station, but the whole area around.

So if it's true, A lot of people could benefit from that extension. And we don't know. So what are you going to do? You're going to go to a specific stop where it is very likely that the people who would be going to MIT or those areas of Kendall Square that would benefit from this extension would alight, and you're going to ask people, would you have stayed on your bus if this bus had continued to MIT and Kendall Square? It's a simple oral survey, yes or no question, one question.

You're going to work in teams of four people. The stop that you're going to station yourself in is shown in figure 3. And you're going to collect data for the AM peak, from 7:30 to 9:30. You pick the day. The teams are assigned on Stellar, so please log into Stellar and see what your team is and coordinate with them to pick a day.

And tell me what that day is, because-- actually, right after class, I'm going to set up a shared spreadsheet that you can all access. And just go into that spreadsheet and pick a day. I'm going to put all the days that are available, and you can say team 1, team 2, et cetera. Make sure that two teams don't go on the same day. We want data from different days.

And you're going to all bring that data together in that same spreadsheet, and there are some questions for you to analyze the data that you collected, all of the class collected together. You're measuring the percent of people who would have stayed on the bus, right? So it's a proportion.

And one submission per team in PDF format to Stellar. This is due March 7, but in order to leave you enough time to do the analysis, the data collection efforts should be done by February 28. So please submit your data by the end of Tuesday, February 28 at midnight, say, or sometime before the beginning of March in the morning, where a person would be trying to analyze your data.

OK, if you have questions, let me know. And if not, have fun. Remember that assignment 1 is due Thursday. Eric?

AUDIENCE: Just the one question: [? is that ?] [? this is ?] going to miss anyone who is transferred to the Red Line to then go to Kendall Square.

GABRIEL SANCHEZ-MARTINEZ: And going back to-- let's see. I forget where I had it. Well, I guess what I-- there was a point I made earlier where we can measure that from automatically collected data, right?

AUDIENCE: OK.

GABRIEL SANCHEZ-MARTINEZ: Does that make sense?

AUDIENCE: Yeah, people who [? car up ?] come from 70.

GABRIEL SANCHEZ-MARTINEZ: So if I see you tapping of the 70 or the 70A, and then I see you tapping at Central Square, I can infer that you were using the service to transfer to Central Square. And then we'll cover ODX, which is an inference model for destinations later in this course.

But looking at the sequence of taps, I can infer-- we can infer-- what the destination of that bus trip was. We can infer that it was the stop that was closest to Central. And later that day, presumably the person who might be going to Kendall Square Station after work taps to Kendall Square. So I might think, oh, he took the Red Line from Central to Kendall. So I don't need to ask those people where they're going. And anyway, they might not care about this extension. So we're going to stand on the bus stop that is after Central Square and see where those people are going and whether they would have stayed on that bus.

AUDIENCE: Is this an actual [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Some people are proposing it. It is a real proposal. The MBTA is a big organization. So I can't say that the MBTA wants to do this or doesn't want to do this. But some people are interested. And it will get looked into. So it's useful.

AUDIENCE: [? Can ?] [? we ?] [? share ?] [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yeah, why not?

AUDIENCE: [INAUDIBLE]

GABRIEL SANCHEZ-MARTINEZ: Yeah. And I guess one other thing that I-- yeah, so we're going to probably make of this like a theme of assignments. So there's going to be another assignment on surface planning, operations planning. So we're going to start looking at this combination of Route 70 and 70A, and we're going to essentially make a thread of this and do some serious planning on some scenarios where the 70 and the 70A could be merged.

And they could maybe be terminated a little-- yeah, we'll make some changes to the service plan under some hypothetical scenarios. And you'll get a chance to do an operations plan on these. And then the last homework will be on policy, so there might be some policy questions that I have in mind about what we could do about service outside, on the outer parts of the 70 and 70A. All right?