Yuma 4×4

Media and Communications

Keynote: Ads and Dart (Dart Developer Summit 2015)

Keynote: Ads and Dart (Dart Developer Summit 2015)


FERHAT BUJUKKOKTEN: Hi,
I’m Ferhat Bujukkokten, and I work in Google
Ads– specifically, we work on front
end infrastructure. And about two years ago,
I got the opportunity to join one of the first
large scale projects to use start at Google. And the project was basically
updating an existing CRM system– a customer
relationship management system. And our goal was to change
the back end in flight– basically, update all of our
back end systems and our RPC layers. And at the same time,
solve some of the issues we were seeing on the front end. So some of the hints that an
existing system needs updating is suddenly the application
starts looking data. Teams– very large teams–
are under high pressure to deliver changes
inside the application, and you see that the visuals
start suffering first. Today, all of us
have a very high bar to meet in terms of making
our UIs pixel perfect, having very good UX,
very good user flows, and delivering very
polished interfaces. So the existing
app was definitely suffering on that side. Also, it’s started
showing performance issues inside the application. And the team just didn’t
have enough productivity to keep up with the changes that
were happening in our business. So for the CRM system, it was
pretty much from the ground up rebuild. And Ted started working on a
project at about the same time, too. What was your
experience like, Ted? TED SANDER: Yeah, my story
is a little different. I didn’t get to start
from a brand new project and start from the ground up. I had this frustration
coming to work every day, and I’m sure many of you
have the same experience. Things were taking too long
to see a change in our system. When we’d make a change, it
would take 10 to 15 minutes before you can actually
see that change on our local environment. You just didn’t really
want to make a new feature because every time you
touch a piece of code, it would affect
four other places. Everything was highly coupled,
it was hard to get velocity, your test coverage
might be iffy, you might have
flaky tests, it was hard to actually know
if something was working the way you expected it to,
there’s all these hidden gotchas in your
code where if you made one change it had unknown
circumstances in the other and you were causing
bugs for your users, you weren’t getting
your stuff out on time. And it just really
created this frustration that I really didn’t like
the work that I was doing. And yet, at home, I had
this awesome experience. I could use Dart, it
had batteries included, there’s all these tools
that I could leverage and I could start doing
awesome projects right away. It was like being
on a rocket ship. It just made my
experience so much better and I loved to code again. It was night and day from what
I was experiencing at work. But something had to give. Something had to change. So I went back to work and I
started analyzing the problem. I was like, how
in the world can I bring that experience
that I had a home and bring it to work and
make the environment better for my team, for myself,
for perhaps others? How do we make the
situation better? And looking at the problem,
it seemed impossible. It seemed ominous. There’s so many projects
we needed to get updated. There’s so many pieces
that needed to change. Stuff that was legacy
wasn’t quite sure how it worked and we were
afraid of updating it. It just seemed impossible. And I didn’t work at
an old world company. We didn’t use a waterfall model. We didn’t do our
requirements out for a year or two before
we even started a project. I worked at Google– a modern,
fast, iterative company that built many of these tools
that I was using at home. If I couldn’t make my developer
experience better at Google, where in the world could I? Did I need to go
work for a start up? Did I need to switch teams? So I chose to do
the Googly thing. I chose to try to make the
environment better for myself, for my team, and maybe for more. Now, I’m not saying that
I was the one person that got everything started. That’s not what it
was an entirely. It was I was a pebble
along with Ferhat, myself, and many other teams at
Google that were getting frustrated with the
environment that we were using. And we started creating
this avalanche. And it’s just kind
of swept us all up. You start seeing so
many teams switch over to Dart and the environment
that it provides. And it’s really this great
story between the Dart team, who really want to build these small
to large humongous apps that are performant,
that are beautiful, that literally help
you code on the web, and these add teams that have
the exact same requirements. We don’t want to have to
worry about our frameworks. We don’t want to have to
worry about our language. We want to have to write our
production code for the problem that we’re trying
to solve, which is allow advertisers
to create their ads and really sell their
product across the web. And in the process, it makes
Google a little bit of money. FERHAT BUJUKKOKTEN:
So we basically had this inherent requirement of
having the teams be productive and very productive. In Google Ads, we have
many mission critical apps. And they start out at a
couple hundred thousand lines, and we have systems that are in
the millions of lines of code. And what makes it even
harder is we have very large and distributed teams. So we have ads teams all
over the world– in Europe, across many time
zones– and basically we had to get productivity
up to a level where we could keep up with
the changes in our business. Google Ads is
evolving very fast. The requirements change. Literally every
month, every quarter, we have to make changes
in our application and add features to support
our users, our business needs. And the needs that
are changing so fast require teams to be
very agile and be able to confidently make changes
on very large code bases. TED SANDER: Yeah, and
it wasn’t like that. Two years ago, it wasn’t
a good situation for us. Some of our projects would
take 15 to 30 minutes before we could actually see
a change on our local machine. And this doesn’t
create good practices. You start writing as much code
as you can before you actually see if it starts working. It’s really hard to make
even simple CSS changes or see how your style flows. Even the simple things
become really tough. And it had some
other problems too. We were finding that the time
that our UX people and our UI people were getting us
mocks for us to implement, it was just taking way too
long to get those mocked and actually get them into
our production system. And it was creating
unhappy users because we weren’t giving
them features that they wanted in a timely manner. It was causing unhappy
managers because they weren’t seeing progress. It was causing unhappy UX people
because they were coming up with all these cool
ideas and we just had to push back on them because
our systems really wouldn’t allow us to make it work. So we started evaluating how
could we make this better– how could we make
our environment this fast and productive
experience for everyone? Make our developers happy
and make everybody else happy in the process. And Dart was really one of the
answers that we liked a lot. We evaluated a lot of
languages, but the secret is that all of them had types. We really didn’t want to try
to use a system within Google that didn’t have types. We have a lot of
help on our code. Even if you use VI
or Emacs, we have tons of systems
that can understand the semantics of
our code and can help us to find bugs
earlier, whether it be a linter or a test tool
that actually can find them or somewhere where
you can just go into a web search of your
code and you can figure out every instance of it. You can figure out patterns
of how it’s being used. You can click on the
type and figure out where it’s being used and who
else might be implementing it or who is overriding it
or whatever it might be. It makes it real easy
to grok this huge code base that we have. You can go in and see. And we can use infrastructure
across the whole system and really understand where
it’s being used within Google. Another feature
that we really liked was Dart is as fast as
JS, sometimes faster. And in the times
where it’s slower, it’s only a little bit slower. It’s very important
for us to get very, very performant systems. We want 60 FPF. It’s a challenge for us,
but that’s what we want. Even on our ad
systems, we want it to be fast,
beautiful, and vibrant and have smooth animations. And we didn’t have that. We weren’t experiencing
that at the time. We were pretty far
away from the metal. It was hard for us
to make changes. Our system before was
trying to protect us against all the different
browser languages. And we had kind of
grown past that. The browsers had gotten better. They were pretty much working
on the same platform now. And there’s all
these things that made it really
hard for us to dive into the internals of what
was happening with our DOM– to make this really
fast paint speed, make our widgets beautiful
and vibrant for our users. A hidden thing that we
got, and you might not think that we need it at
Google, was the package manager. And it’s not really because we
need go download code from pub and use it in that way. But we got a lot
of hidden benefits from using this common
infrastructure within Google. Dart had batteries included. The libraries would
work together. We didn’t need to try to figure
out which keyboard library we were going to use, which
stream library we were going to use, what module loading
library we were going to use. It was just there. It was inside Dart. And it gave us
all these patterns that we could reuse
over and over again. And all of a sudden our code
started looking really similar. You see the future pattern
in a lot of our code. You see streams in
a lot of our code. It has actually inspired
other libraries. It’s pretty cool. And it also allowed us to start
using more external libraries that we found. We could bring in
external libraries and we could just
start using them and they would work
within the ecosystem. We didn’t have to
worry about them conflicting with other libraries
or not working in our framework or not having types so
that we could easily understand what was going on. And a great story of
this is we weren’t really happy with the mocking
story the Dart provided us. It uses string-bass mocking. It wasn’t really
good to refactor it because you might
refactor the method but you didn’t actually refactor
the string that was trying to make sure that it matched. It didn’t have auto
complete a lot of the times. We weren’t in love with it. And we could’ve just went
to the Dart team and said, this isn’t good enough for us. Please rewrite it all
and give it to us. Spend some cycles and ignore
all the cool other stuff that you’re doing for
us and everybody else and please give us a
better mocking library. But instead, we found
in external one. We found Dart Mikado. And we could just use
that– we could bring it into our system and
everything just worked and there is nothing we had to
do to make it so that we bring in the libraries or whatever. This ecosystem was just vibrant. We could start
leveraging external. We could start giving to
the external community too. And this wasn’t
actually always true, but we actually used the pub
system within Google also. And before we actually
had some of our own tools and we were actually
envious of you guys– the external developers. Because you had pub serve
and you had pub build. Everything just worked for
you, and there is no fires or there wasn’t glue
sticking everything together. And sometimes it worked
and sometimes it didn’t. And now we actually use pub
build and we use pub serve. And it allows us to have a
really fast refresh cycle to leverage all the
tools that you get. We get a leverage that
test runner that’s coming that you guys saw. And it’ll just work for us. We don’t really have to worry
about building our own tools to do all that cool stuff. So really, this is a story of
the synergy between the Dart team and us– of giving
us the tools to become powerful and really worry about
only the problems that we need to solve instead of trying
to solve all these language and framework problems. FERHAT BUJUKKOKTEN:
So we wanted to share some of the
experiences we’ve run into in using Dart
in our projects over the past couple years. And also point to
some of the strengths that helped us
solve these issues. So one of the big items
is always performance. When I joined the CRM project
and we were just starting out, I had a meeting with
a project manager and he started demoing the
existing application to me. And some of the views–
some of the reports– were so slow, in the
middle of the meeting, he just stopped demoing it. And he started talking
about the future what he’d like to see in the new system. So applications
never start slow. They start getting
slower and slower as the application is growing. You’re adding a
lot more features. You’re making changes. You’re adding abstractions and
more layers to your system. And sometimes it
feels like there’s this black hole in
the browser and all your CPU cycles are disappearing
into the black hole. And so one of the
issues we ran into was when you closely
monitor a production system, you get your p50 numbers–
50th percentile performance, 90th percentile performance. And looking at enough systems
and enough applications out there, you have a good idea
of what the ratios should be. So p50 for us is
pretty much working in Mountain View with
excellent networks and being close to
our data centers. And p90-95 is people are on
slower connections– maybe they’re on older hardware. And the numbers always reflect
that no matter which system you look at– if you look at
Gmail, you look at calendar– there are constant factors. And in our system,
after we started testing it– that
was written in Dart– we basically saw that
in p90-95 numbers. We had a big drop
off in performance. For example, we had
these giant tables with 20 columns and
thousands of rows. And when you sorted the
columns, it should be instant. And on p90th, we were seeing
these really strange and big numbers that were beyond–
we couldn’t explain really with the hardware differences. So we turned our
attention to memory because the logs also showed
some of these applications, they’re mission critical. People sit down in the morning
and use them all day long. They CRM map. They make calls,
they log meetings, they look at sales
pipelines all day long. And what we saw is
on p90th and 95th, we saw very high memory usage. So we said, OK, maybe
that could explain what’s going on in performance. So we ran some numbers. We basically started stress
testing the application and started running the
browser slowly out of memory. So what happened
was as soon as we had about 660 to 670
megs of memory use, we saw a very sharp
decline in performance. Essentially, the app
would be five to six times slower under high
memory pressure. And when you build a large
app and you build thousands of lines of code and
300 to 400 libraries, it’s very hard to go back and
find where the bottlenecks are. But with Dart, we had
these wonderful tools. Observatory really helped us
diagnose some of the issues, see where we had services
that we’re global. We were subscribing to streams
and never releasing them. And as the user was
going through their day, essentially we were
holding onto memory. So tools like
Observatory allowed us to automate a lot of this and
make sure that going forward we have both memory and
performance under control. So yesterday you heard the
protocol for Observatory is going to be published. And this will allow you to
basically run your tests, go to a blank page, go to your
view, go back to a blank page, and then talk to
Observatory and say, do I have any
controllers– any views, if you’re using MBC–
still on the heap? So you can easily build
these into your tests and make sure whenever
people add features– and once you have hundreds
of engineers adding features, you want to make sure
that these problems don’t get worse in your system
as your application grows. So today, for a very large
business application, we have very stable and
low memory use using Dart. So typically at p50th, we’re
in the 100 to 150 range. And p90th, we’re staying
well under 300 megs. So the tools that Dart provided
were really key for us too, not only in our
daily workflow to be able to just pull up
Observatory and use it to make sure we didn’t
have any issues regarding strong references that
we were holding on, but these tools also
allow you to build them into your systems that
monitor your health. And another big
requirement we have in Ads is that we need to scale
to very large teams. So when you put together a
large number of applications in one big system, sometimes it
feels like it looks like this. Essentially, the bits
and pieces work together. But it’s not exactly a harmony. So with Dart, as Ted said,
batteries are included. It shipped with wonderful
libraries from day one. And the libraries are
consistent across. So if a library had to
use a certain pattern, it was iterables or streams
and it was consistently so. We didn’t have one library
in the system calling it push or destroy and
another library calling it add and clear. It was these patterns
that are nicely followed by not just the
dart team but all of you out there publishing
pub packages. The libraries look very
similar and work in harmony. I can easily read
code from a team I had no contact with before. If they’re using
Dart, it just feels natural to reuse
their libraries. And that sort of
helped us become– although we have huge
number of engineers, we became a smaller team
that reused more code. And some of the
libraries we built we were able to open source. For example, in
the business app, we had needs for showing
pipeline charts but 50 of them on a screen for each customer. You would just see
your sales pipeline. And we needed to do
that in x milliseconds. So it was a good experience. So we took these three
and we did a port. It’s called charted. So you can find it on GitHub. It’s basically a
nice charting library and we shared that with
the Dart ecosystem. Some of our RPC libraries
we were able to share. And a lot of the dart
inspired all the teams to follow the same patterns. Our RPC back end, when we say,
give me a customer record, we literally get the stream. We say, listen on the stream. Send the query,
listen on the stream. And then whenever
there’s an update, our RPC layer just pushes
it through a stream. So it’s allowed us to create
much, much nicer libraries. TED SANDER: So like
I was saying before, we had this is problem of
our apps were really large, but they were really highly
connected and tightly coupled. It was hard to even
get our head around how we were going to start
updating parts of it. Not all our teams had
the luxury of a team like Ferhat to just do
an RM-RF like Dan said and just start from scratch. We did actually
have the requirement that we did have to
build up some of the apps and slowly start iterating
to get a better system. And so how we did this was we
started modulizing our apps. So even though to the end user
it looks like one large app that they can use,
internally, it’s actually a couple smaller
apps built together to become one coherent one. And this is very
important to us. Because we can’t allow
the user to say, well, if you want to do
search ads, you need to go to this
application over here. If you want to do
YouTube ads, you’ve got to go to a different one. If you want to have display
ads, well, that’s another URL that you need to go hit. It was very important
that we had one stop shopping for advertisers. But yet, we found
that if we tried to do this all in one large app,
our team’s velocity grounded to a halt. It was really
hard to add more engineers on this humongous
app and allow them to iterate quickly and
start doing new stuff and to build out new
features and really get the velocity that they
crave and they desire. So how did we do this? Well, we started
splitting them up, and yet we used this
very simple pattern to make sure that they could
still talk to each other. And why do we want
to keep it simple? Well, we want one
very simple interface because all these
modules will need to talk to each other using it. And if it’s complicated and
we have to change it often, there’s a chance that we’re
going to break all the time. And we really don’t
want that to happen. So if we keep it
simple yet powerful, we don’t want to
change it too often and yet everybody
gets the power of it. This actually turns out to be
like six lines of JavaScript that allows us to talk
between all these apps. And the cool thing
about it is we can have different technologies
for each one of these apps. We could have an
app written in GWT, we could have an
app written in JS, we could have an app written
in Flex if we wanted to. We don’t, but we could. As long as it talked
JavaScript, we could actually have them
talk in between each other. Yet, it was important for us to
have this consistent API layer. We needed the data
consistency always. You could imagine that if our
data ever gets inconsistent, it is a nightmare to make
sure that it’s cleaned up at the end of the day. Migrations are a pain, and
we do not like doing them. Also, if there’s a
bug in the system or you accidentally introduce
some data the shouldn’t have, it’s possible to cause millions
and millions of dollars to be lost very easily and
it’s very scary sometimes. So if we had this
consistent API layer, we could make sure that
the data was always saved in the same way. So all the apps, even though
they were separate apps, we’re still talking to the back
end in a consistent manner. So let me give you
an example of this– and I’m actually going
to do a live example. So this is AdWords. Many of you might not
actually recognize it, but if you were an ad agency, it
would be very familiar to you. And on the left and on the top,
that’s actually separate apps. And they’re GWT apps right
now, and what we actually had before. But this app in the
center right here is actually written in Dart. We could allow us to do
business logic– covers, all that good stuff– and it
could interop between the two systems very easily. So it allowed us to
iterate– our team to iterate– and get some
Dart app code out there without rewriting everything. Another benefit is these
charts were pretty complicated. We didn’t want to write
them from scratch. So we could leverage
the JS library we had a Google to
actually provide these charts for us, allowing
us to not have to rewrite every library if we chose to. And it allowed us to keep
the animations and all those cool things, but yet still
have all of our business logic and all the other UI
elements in Dart still. Now, the crazy part– if
we want to go even deeper– is this app is now a GWT app. So now we a GWT app
inside of a Dart app which is inside of a GWT app. We’ve got to go deeper, right? So it really allowed us to
interate and not replace everything if we chose to. Sometimes we could. We could do what Ferhat’s team
did and just burn everything and start anew. But it also allowed us to
iterate and make changes individually and
allow teams that wanted to velocity without
sacrificing all the hard work and app that they had currently. So the strategy for this is
really divide and conquer. Even if you could start
over, it’s still a good idea to keep your app modulized. You hope your app can
scale from a team that’s two or three people to
hundreds and hundreds. And if you keep them
modulized, you can do that. You can keep the problem
smaller and allow them to think about it. FERHAT BUJUKKOKTEN: So
finally, before Kevin throws me off the stage– we’re a
little over time– we ended up with a wonderful system. So in six months,
we built the app and shipped the first version. And CSAT scores
more than doubled. And we had a really happy team. So the developers were happy. The users were happy. And some of the
experiences we had, they were so successful
Ads decided basically to invest heavily in Dart. And many of the teams
are now coming online and they’re starting to
build their apps in Dart. We’d be happy to talk
to you afterwards. And thank you.

Leave comment

Your email address will not be published. Required fields are marked with *.