Bangalore Configuration Management Meetup Group- (First Meetup- March-02-14)

We had our first CM meetup today at the Square Cafe Coffee Day; We booked a Huddle room for this meetup. It is well equipped with projector, wifi etc. Few folks have RSVPed yes but as expected they did not turn up :) but anyway we had a great session. There were only four of us for the first meetup. Participants: 
1) Bernard L'Allier- RackSpace Inc 
2) Roopesh Kumar- Streamnet Cloud 
3) Krishna Gowda- Streamnet Cloud 
4) Subhankar Sengupta- Sabre Inc.    

We waited for 15 minutes for others to join and then finally started by 2.30 PM. After the introduction was over, I did a presentation about "Configuration Management as a Practice". This was more about looking at the whole picture of CM practice and not just System/Service provisioning and deployment. Spoke more about the control tier API. Control tier includes Orchestration, Dispatcher and Scheduler. Then covered the provisioning tier with example of Puppet/Chef/Salt etc. Touched upon importance of version control and artifact repo for infrastructure configuration components. After that Bernanrd shared his insight about the CM practice in Rackspace and specifically discussed few case studies for CM. Apart from that Bernard shared his vision for Rackspace in India with us. Roopesh and Krishna shared how CM practice with a introduction of tool like puppet/chef could make their support echo system more effective. 

Finally we did a Live Workshop on Puppet and we covered following topics (Hands-On) -
1) Puppet Installation.....
2) Puppet Directory Structure....
3) Apache Deployment with Puppet...
3) Script Deployment with Puppet...
Meetup ended in a very positive note and all of us took good amount of learning back home :) We decided to meetup again in near future (Date will be announced) and will uphold our promise :)- No Sponsor, Zero Marketing, Strictly Open Source and Vendor Agnostic :)

Apache and Nginx

I recently came across a blog  that says Nginx performance is better than Apache because, “Nginx doesn’t create a new process for a new request. Apache creates a new process for each request” This is not completely true, Apache MPM has two models, 1. Prefork (Above statement is applicable for this model) 2. Worker The worker MPM uses processes with many threads for each process. Each thread handles one connection at a time. This model is suitable for high performance web servers. I have not worked on Nginx and but managed high few high traffic website with Apache (Tuned) and lighthttpd quite well. I wanted to learn more about the performance advantage that Nginx has over Apache or lighthttpd. Is there any study/load testing done to prove Nginx could perform better than Apache (Tuned) for high traffic website? Please back your statement with data. (In Data we trust rest is ;)
In response to Event Driven Arch of Nginx

Agree Buddy :) Nginx is event driven and asynchronous from design perspective and I guess this very helpful for more requests that are long-running but of a small size (Example Mobile)
As ONESTONE says:
"Nginx uses the Reactor pattern. Basically, it's single-threaded (but can fork several processes to utilize multiple cores). The main event loop waits for the OS to signal a readiness event - e.g. that data is available to read from a socket, at which point it is read into a buffer and processed. The single thread can very efficiently serve tens of thousands of simultaneous connections (the thread-per-connection model would fail at this because of the huge context-switching overhead, as well as the large memory consumption, as each thread needs its own stack)"

But Apache has a fix around it- Event MPM (http://httpd.apache.org/docs/2.2/mod/event.html)
As I know this fix is experimental and has issue with https :(
But again as per Kevin, Apache could outrun Nginx when "you are running pure PHP content on a given server, Apache seems to still be the best bet for the job."

http://www.eschrade.com/page/why-is-fastcgi-w-nginx-so-much-faster-than-apache-w-mod_php/

I would design this in following way,

* CDN (CSS, JS and images)

* Run PHP from Apache (Tuned)

Again as I mentioned before I don't know Nginx (Never worked on it) so idea is to LEARN and EVOLVE. Leverage Nginx when required and possible.
I never carry any preconceived notion/baggage of dogma :) just a student of life/events :)

My thought on battle for platform

A decade back OS (operating system) was the platform for IT companies. So Microsoft, IBM, HP, Sun all wanted to capture the OS market. Idea was- one that controls the platform will control the business app as well.

But as Linux/Open Source matured with time, control of OS became irrelevant. Now anybody could have or build an OS. Android is an example.

 I sincerely believe the new battle for platform will be for search engine/search technology. 

(I consider Google and Facebook as a new platform and more significant than OS in today's context)

In a nut shell- There is enough content already available and daily/hourly new contents getting generated;  need is to SEARCH, CO-RELATE, ANALYZE  and DELIVER the content to relevant seekers.

Please note content could be anything, even a business app, code, API, product review, hotel room, image, pet and even your life partner ;) could be content and the seeker does not necessarily need to be a human, even a system/software or business could be seeker.

The battle for new platform has already started. I could count three as of now:)

Google, Facebook and Microsoft ;)

Tuning JDBC- Part 1

Before we start please note all R&D should be done in integration environment and not in production , expectation is all bugs/app leaks are closed before you move to production.

Sometime (particularly when you have too large DB/too many DB connected to a single jBoss instance) jBoss throws   No Managed Connections available and timeout error.

This means that all connections to the database are saturated or a thread has timed-out waiting for a connection to open up. Now this could happen due to multiple reasons.

1.  Predominantly we forget to close connection in code 

Solution- Stop leaking any open database connection in app. Go to $deploy/jbossjca-service.xml

and add  <attribute name="Debug">true</attribute>, this will allow jBoss to throw a message like

“Closing a statement you left open, please do your own housekeeping”

2.   Connection pool size

Solution- Increase your connection pool size in datasource file

<max-pool-size>20</max-pool-size>

(Default is 20)

3.  Thread time out

Solution- Increase thread wait time just add below script in  your datasource file

<blocking-timeout-millis>xxx</blocking-timeout-millis>

Note: xxx should be much more than timeout value

4.  Old JDBC Driver

Solution- update driver, it is a JAR file, download and copy into  JBoss  jboss-as/server/myinstance/lib

Happy Tuning

Open File Descriptor- Happy Tuning

What is a File Descriptor in Linux?
As per dictionary.com literal meaning of descriptor is an element or term that has the function of describing something specific.
In Linux Operating System a file descriptor stand for an interaction between a process and an opened file, while an open file object has information related to that interaction.
Numerous processes could simultaneously open the same file. In that scenario, the filesystem assigns a separate file descriptor to each file, along with a separate open file object.
Under Linux OS-  fd indicates the file descriptor of the opened file. There is a default limit for maximum number of file descriptors that an individual process could open and in Linux it is 1024.
Kind of less :(
At times this max limit could be a bottle neck for some java apps so we use ulimit to tune it.

ulimit -aS displays the soft limit, and ulimit -aH displays the hard limit

Example:

# su - httpd

# ulimit -Hn

# ulimit -Sn

Tuning Open file descriptor max limits

# echo “fs.file-max=95000″ >> /etc/sysctl.conf

# sysctl –p

 # echo “* hard nofile 95000″ >> /etc/security/limits.conf

# echo “* soft nofile 1024″ >> /etc/security/limits.conf

# echo “session required /lib/security/pam_limits.so” >> /etc/pam.d/login

#ulimit -n unlimited

:) Happy Tuning

Appdynamics Lite- A free APM

Kind of smitten by Appdynamics Lite, cool features and

it is absolutely free of cost. I just tested Java transactions

for a simple webapp and very much impressed and voila

Appdynamic Lite has a .Net version(Free) as well.

Appdynamic Lite Introduction, it has 2% CPU overhead,

10 MB memory footprint and <5 ms latency for user requests.

 

I need to explore this product more and will share finding.

Watch out for next.

 

 

Java Thread dump and Heap dump

I found many people in industry assume Java Thread dump and Heap dump as same.

A thread dump is a complete list of Java threads that are presently active in a JVM. This is good for understanding behaviour of a Java application at a given time. Most of the time app developers will request app infra team to get a thread dump for their analysis. (e.g. deadlock).

 

How to take thread dump?

#jstack <pid> >> threaddumps.log

A heap dump is a dump of the Java heap memory. This is useful for analysing what use of memory that Java app is making in JVM, help in diagnosing some memory issues. (e.g. memory leaks)

 

How to take heap dump?

#jmap -dump:format=b pid