Wednesday, 22 June 2011

UnderStanding Systemd Part -II

This Post is from my understanding of the original post by leannart poettering on systemd :- 

At this point may be we should look back and analyse how the init scripts work, and have worked over the past so many generations, and may be draw a parallel with kde's style of booting up.
SystemV init ----> Upstart ----> Systemd

SystemV's Approach :-

Finding the dependencies between services and starting them according to a topological sort is inherently serial. You could start daemons that don’t depend on one another at the same time, but if two daemons both depend on dbus, dbus must start first. This creates inevitable bottlenecks which slow down the boot up.

Upstart's Approach :-

“All jobs waiting on a particular event are normally started at the same time when that event occurs”. This means that all services that depend on dbus are started immediately after dbus has finished starting. Although this certainly improves boot times by a fair margin (Ubuntu boots very quickly nowadays), the bottleneck is still there.

Systemd/launchd's Approach :-

What Happens is that for every daemon the resources gets allocated right at the start,(resources here implies, the sockets and the ports), while keeping the mapping from resources to the daemons, it is pretended that everything has been started, while loading all the daemons simultaneously. If a daemon requires a socket that belongs to a daemon that has not finished starting yet, the requests are buffered and ordered by the kernel and the requesting daemon blocks, waiting for an answer. Finally, when a daemon finishes starting up, it gets all the requests buffered by the kernel without any fuss, as if it had just received them.

I am not too sure, but from the look of things, it seems that kde's model of start-up is pretty much similar to either Upstart's or SystemV's approach, starting one process after another serially. Analysing my system boot charts, it looks more close to Upstart's approach. Here the order of execution is :-

KDE Display Manager ----> startkde script ----> (1). kdeinit (starts various services)  ---->  klauncher + kded + kcminit
                                                                              (2). ksmserver


Image Showing System Processes of my system with init script being high-lighted .
To achieve something close to systemd's model, the focus area should be the tie-up between kdeinit script and its subsequent sub-processes, and how we can parallise the process calls instead of the current serial inter-connections, between the processes. Apparently this approach should be highly feasible for an enviro like kde/gnome. But editing and implementing all these scripts is still going to be a Herculean task.

Friday, 17 June 2011

Understanding systemd part-I

This Post is from my understanding of the original post by leannart poettering on systemd :- 

From the look of things systemd just looks like a fancy init system.

May be at this point, I must get more familiar with init system, to understand more about systemd. init has PID 1, and so obviously it gets started first by the kernel, before all other processes. It is the parent process to all other child processes. Apart from this very important function, init script performs the central task of bringing up and maintaining user-space during boot. init script is much faster than its venerable predecessor sysvinit. For a faster system, 2 main things are required.
(1). Start less ---> Starting less means starting fewer services or deferring the starting of services until they are actually needed. For example a printing service may not be required immediately after having system on, whereas, there are some services where we know that they will be required sooner or later (syslog, D-Bus system bus, etc.). There would be no need to start many services, until directly called by the user, or its API is required by some-other service.
(2). Start More in parallel ---> Starting more in parallel means that if we have to run something, we should not serialise its start-up (as sysvinit does), but run it all at the same time, so that the available CPU and disk IO bandwidth is maxed out, and hence the overall start-up time minimised.

Lets see how programs start, with an example :-

--------------X--------------------------------------------------------- DBus service
---------------|------------------------------------------------- --X--- HAL service
--------------X---------------------------------------------------X---  syslog
               |                                                    |
               |                                                    |
Avahi(dependent on dbus and syslog)---------->livirtd (dependent on HAL and syslog) & also on Avahi

So for livirtd to start, it would have to wait for HAL and syslog to start, and additionally wait for Avahi to start too, hence livirtd can't start until all these services have started, and hence the incumbent delay.

Parallelizing Socket Services :-
In-order to get rid of these synchronising and paralleling delays, we have to understand what is required by one process from another. Usually that is an AF_UNIX socket in the file-system, but it could be AF_INET[6], too. For example, clients of D-Bus wait that /var/run/dbus/system_bus_socket can be connected to, clients of syslog wait for /dev/log, clients of CUPS wait for /var/run/cups/cups.sock and NFS mounts wait for /var/run/rpcbind.sock and the portmapper IP port, and so on. Now if we can make this socket appear before the entire daemon gets executed, and link it with the service, we can significantly reduce boot time and start more processes in parallel. We can create the listening sockets before we actually start the daemon, and then just pass the socket during exec() to it. That way, we can create all sockets for all daemons in one step in the init system, and then in a second step run all daemons at once. If a service needs another, and it is not fully started up, that's completely OK: what will happen is that the connection is queued in the providing service and the client will potentially block on that single request. But only that one client will block and only on that one request. Also, dependencies between services will no longer necessarily have to be configured to allow proper parallelized start-up: if we start all sockets at once and a service needs another it can be sure that it can connect to its socket.

## not clearly understood this part ##

Basically, the kernel socket buffers help us to maximise parallelization, and the ordering and synchronisation is done by the kernel, without any further management from userspace! And if all the sockets are available before the daemons actually start-up, dependency management also becomes redundant (or at least secondary): if a daemon needs another daemon, it will just connect to it. If the other daemon is already started, this will immediately succeed. If it isn't started but in the process of being started, the first daemon will not even have to wait for it, unless it issues a synchronous request. And even if the other daemon is not running at all, it can be auto-spawned. From the first daemon's perspective there is no difference, hence dependency management becomes mostly unnecessary or at least secondary, and all of this in optimal parallelization and optionally with on-demand loading. On top of this, this is also more robust, because the sockets stay available regardless whether the actual daemons might temporarily become unavailable (maybe due to crashing). In fact, you can easily write a daemon with this that can run, and exit (or crash), and run again and exit again (and so on), and all of that without the clients noticing or loosing any request.

Parallelizing Bus Services
Modern daemons on Linux tend to provide services via D-Bus instead of plain AF_UNIX sockets. Now, the question is, for those services, can we apply the same parallelizing boot logic as for traditional socket services? Yes, we can, D-Bus already has all the right hooks for it: using bus activation a service can be started the first time it is accessed. Bus activation also gives us the minimal per-request synchronisation we need for starting up the providers and the consumers of D-Bus services at the same time: if we want to start Avahi at the same time as CUPS (side note: CUPS uses Avahi to browse for mDNS/DNS-SD printers), then we can simply run them at the same time, and if CUPS is quicker than Avahi via the bus activation logic we can get D-Bus to queue the request until Avahi manages to establish its service name.

Apart from these services, filesystem jobs also have to be parallised, but I am not going to read much into it, as the main focus of my project, should lie in maintaining Bus and Socket services.
Technorati Tags: , , , ,

Thursday, 16 June 2011

UnderStanding KDE Launch Sequence

Its time to begin some serious work, I am already behind schedule, I will have to choose an area to work-on and then move-on to doing some instrumentation/benchmarking to see where the most time is spent, and where the efforts have to be focussed. I have decide to take a look at complete launch sequence right from KDM to the Plasma Shell, with Akonadi Application. My modest assumption, at this stage is that the bulk of the time for all the applications is spent on the transfer from one process to another, from startkde/kdeinit/kded etc. and their tie-up together, maybe we could unify this a bit more. Maybe also add some timing instrumentation/debug output, to see where time is spent.

Having a more detailed look on the entire start-up sequence :-

KDE Display Manager 
                |
                |
               V
   startkde script (executes 2 main functions) ----> 
(1). kdeinit (starts various services)
   dcopserver + klauncher + kded + kcminit
(2). ksmserver

Process 1:-
The user gets authenticated at start-up by entering the user-name and password at KDE Display Manager (KDM). KDM then executes startkde script, this scripts performs two vital functions, and calls the following scripts :-

LD_BIND_NOW=true kdeinit +kcminit +knotify
Starts the kdeinit master process, which in-turn starts all other KDE processes. The arguments behind kdeinit are the names of additional services to be started. The + indicates that kdeinit needs to wait till the service has finished.
                ## Haven't clearly understood the script ##

kwrapper ksmserver $KDEWM 

Starts the kde's session manager. On startup the session manager starts auto-start applications and it restores applications from the previous session. The session manager determines the lifetime of the session. When this process exits, the user is logged out.

Process 2:-
kdeinit master process begins after the startkde script gets executed. kdeinit can start normal binary program files as well as kdeinit loadable modules (KLMs). KLMs work just like binary program files but can be started more efficiently.

for example :-
> ps aux
waba     23184  0.2  2.1 23428 11124 ?       S    21:41   0:00 kdeinit: Running...
waba     23187  0.1  2.1 23200 11124 ?       S    21:41   0:00 kdeinit: dcopserver --nosid
waba     23189  0.2  2.4 25136 12496 ?       S    21:41   0:00 kdeinit: klauncher
waba     23192  0.7  2.8 25596 14772 ?       S    21:41   0:00 kdeinit: kded
waba     23203  0.8  3.4 31516 17892 ?       S    21:41   0:00 kdeinit: knotify

kdeinit in the first line: Running... indicates the master kdeinit process. The other processes listed are programs started as KLMs.

Process 3:-
BackGround/Subprocesses :-

1. dcopserver 
# confusion? - is it any different from the dbus daemon that starts on system booting ??? #
 The doci says "dcopserver is a deamon which provides inter-process communication (DCOP) facilities to all KDE applications." Which is pretty similar to dbus-daemon, which provides inter-process communication (IPC) facilities to all KDE applications as well as several other system components, such as HAL, network manager, power manager and various non-KDE desktop applications.

2. klauncher
klauncher is a deamon which is responsible for service activation within KDE. It operates in close connection with the kdeinit master process to start new processes. KDE applications communicate with klauncher over dcop in order to start new applications or services.

3. kded
kded is a generic KDE daemon. It has the ability to load various service modules and run these in the background. The Service Manager in the Control Center can be used to monitor the status of the service modules and to disable certain services.

4. kcminit
kcminit executes initialisation services during startup. Initialisation services are specified in the .desktop files of applications or services via the X-KDE-Init line. Initialisation services are typically used for initialisating hardware based on user specified settings. kcminit --list can be used to show all initialisation services and kcminit can be used to execute a single service explicity. This can be useful when investigating start-up problems.
## More Understanding required ##

Process 4:-
ksmserver
ksmserver is KDE's session manager. On startup the session manager starts auto-start applications and it restores applications from the previous session.

X-KDE-autostart-condition=ktiprc:TipOfDay:RunOnStart:true
Whether to auto-start an application can be conditional upon some configuration entry determined by the X-KDE-autostart-condition entry in the .desktop file. The KDE session manager also restores one of the previous sessions. A session contains of a collection of applications as well as application-specific information that reflects the state of the applications at the time the session was saved. Sessions are stored in the ksmserverrc configuration file and contains references to application-specific state information. The application specific state infomation is saved in $KDEHOME/share/config/session.

For example, if ksmserverrc contains:

restartCommand1=kwin,-session,110a0a0002000106760689100000019360000_1083342151_608625
restartCommand2=konsole,-session,10378ac070000108334204700000277690000_1083342151_552722

Then the application specific state information for kwin and konsole can be found in

$KDEHOME/share/config/session/kwin_110a0a0002000106760689100000019360000_1083342151_608625

and

$KDEHOME/share/config/session/konsole_10378ac070000108334204700000277690000_1083342151_552722

respectively.


The state information of kwin contains the location of the application windows of all the other applications in the session.

At this point the order of execution with respect to ksmserver is not very well understood by me, like what happens immediately after the startkde script, if the ksmserver script gets executed alongside, kdeinit, then the values returned by ksmserver would have to communicate in some-way with the klauncher and kded.
             
Technorati Tags: , , , ,

Sunday, 29 May 2011

Season Of KDE

Having not got selected for this year's Google Summer Of Code, I applied for Season of KDE. Season of KDE (SoK) was set up in 2006 to provide some of the benefits of Google Summer of Code to those students whose projects did not get selected. Season of KDE provides students with experienced mentors and a well defined project, just like Google Summer of Code. SoK does not provide payment to students. Season of KDE doesn't provide the same benefits as Google Summer of Code, but it offers valuable mentoring, along with a vibrant and friendly environment, its more about Passion and of-course a cool T-Shirt. SoK allows us to work on a Cutting edge Project, and helps to take first steps developing KDE software and becoming worthy members of the KDE community. Each SoK student works on their chosen project with a mentor from KDE with experience in that area to help and guide them.


I too applied for Season of KDE. The application procedure was very straight-forward. Soon after getting the --> "sorry we couldn't select you, mail" from Google. I received a mail from a small Google Summer Of Dissapointment community. And came to know about Season Of KDE. Even students have to eat and so SoK participants often have other jobs and can only work on their projects part time. As a result, SoK projects may have smaller scope than Google Summer of Code projects or happen over a longer period. KDE benefits from new additions to our software and our community, and students get a SoK t-shirt, a certificate, some Google goodies and a great experience. SoK can also be a springboard to future Google Summer of Code success, with several past SoK participants going on to secure Google Summer of Code acceptance. Equally, SoK has provided opportunities for students to continue a Google Summer of Code project from previous years.

Having filled out the initial details on Lydia's Blog.
http://blog.lydiapintscher.de/
I received a mail next day, about my Project Details, I had earlier very vaguely suggested a project on Kde Speed Optimisation.

Project : Speed Optimisation of KDE start-up time.

I was Bombarded with a lot of well directed suggestions like, :-

: A possibility would be to help the Platform modulation effort to cut down dependencies.

: Another could be to reduce the number of dependencies that are loaded during startup. That is to convert some   of the static libraries and see if we can load them Dynamically, that may have slight performance issues, but that would greatly reduce start-up time.

  And finally I hope to implement the project as Tom Gundersen suggested :-

: Having a look at how systemd has improved startup speeds of system services, and try to do something similar to kdestartup. The idea is to allow sockets/dbus activation to synchronise daemon startup, rather than explicitly calling one daemon after another is up.

Idealy something similar should be possible (and much easier) for KDE apps/daemons as most desktop things are already able to do dbus activation. The idea would be to start all apps/daemons/services simultaneously as soon as the dbus session bus is running, and if one app needs a service it will block in the call to dbus waiting for it to start. As much as possible everything should be in the first autostart phase.

: The way to go here would to use bootchart and a profiler to find out where most of the time is spent, and then making that code faster. systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic.


It looks like these are going to be a great summer, I am so very excited. I hope my project would get selected.

Technorati Tags: , , ,

Friday, 13 May 2011

FireFox Tricks, Linux FunFacts & some interesting Doodles

Haven't posted for a while now, for a variety of reasons.. ranging from frustating blogger maintainence issues to depressing exams, practicals and vivas... So, I am back again presenting to you, a few fire-fox tricks that I came across while googling .... they look pretty cool, so just try them & dont forget to comment with your favourite firefox trick....

Type the following addresses in the address-bar, and see the results :-

chrome://global/content/alerts/alert.xul
IT SHOWS Dancing Firefox.

chrome://browser/content/browser.xul
IT Opens another Firefox inside a tab in the the existing Firefox window.

chrome://browser/content/preferences/preferences.xul
IT Opens the Options dialog box inside the Firefox tab.

chrome://browser/content/bookmarks/bookmarksPanel.xul
IT Opens the “Book Marks Manager” inside a tab in the Firefox window.

chrome://browser/content/history/history-panel.xul
IT Opens the History Panel in the Firefox tab.

chrome://mozapps/content/extensions/extensions.xul?type=extensions
IT Opens the Extensions window in the current tab.

chrome://browser/content/preferences/cookies.xul
IT Opens the “cookies window” inside a tab in the Firefox window.

chrome://browser/content/preferences/sanitize.xul
IT Opens the “Clear Private Data” window inside the current tab.

chrome://browser/content/aboutDialog.xul
IT Opens the “About Firefox” Dialog box inside the tab.

chrome://browser/content/credits.xhtml
IT A scrolling list of names. The one’s who we must thank for creating Firefox

linux funfacts :-
(1). Linus' (The writter of the linux kernel) favorite programming editor is                   Microsoft Notepad.

(2). Linux® is commonly confused with LINUXOS,released by Linux Incorporated.

(3). Steve Ballmer (Microsoft) has called Linux a cancer on mankind.

(4). Linux® is a compound abbreviation for the full name: Linu Christ. In fact,
      Linu Christ is believed to have run the Church of UNIX. Critics such as Pat
      Robertson contend that the X crosses out Christ.

(5). The bit-system was originally invented to make Linux®'s kernel-versions
       countable. Right now there's 64bit, since there are 2^64 different kernels.
       And the saga isn't even close to coming to an end.

(6). The original kernel source was written down in four notebooks.


A few interesting Google Doodles, I came across from Google India and Google World :-

 




 




  
With the last one being, Google's first ever Doodle.