 |
 |
We should have 300 max connections for hpsuser now which is used for the read only recon jobs. With changes I made to how the connections are managed internally, that should probably be good enough for now. If we actually start to create > 300 HPS jobs at once in Auger, then let me know and we can look at implementing a retry algorithm.... > On Dec 28, 2014, at 7:54 AM, Nathan Baltzell <[log in to unmask]> wrote: > > An additional option is to test the connection and wait+retry (N times) if it's > not good before failing. That could work if a single job's connection duration > is short and we still have max_connections problems until the server is upgraded. > -Nathan > > >> On Dec 27, 2014, at 10:50 PM, "McCormick, Jeremy I." <[log in to unmask]> wrote: >> >> Hi, >> >> I made some changes to the conditions system so that the connection should not be left open during the job anymore. >> >> The related JIRA item is here. >> >> https://jira.slac.stanford.edu/browse/HPSJAVA-370 >> >> Can you confirm this by trying to run the batch jobs on Auger? >> >> If 100+ jobs actually initialize at the same time, then you might still see the max connections problem. That will have to be fixed on the database configuration side. I have requested that max connections be increased to our maximum allocated batch slots on Auger, but this might not get done until after the shutdown, as it requires the database administrator to change. >> >> —Jeremy >> >>> On Dec 26, 2014, at 7:26 AM, Graham, Mathew Thomas <[log in to unmask]> wrote: >>> >>> >>>> On Dec 25, 2014, at 10:08 PM, McCormick, Jeremy I. <[log in to unmask]> wrote: >>>> >>>> I believe that should be tunable on the MySQL db via max_connections setting, but would need to talk to the admin about changing it. Around 100 is probably the default. How many simultaneous connections would we need? >>>> >>>> Also, it should be possible to close the connection in the db manager after conditions setup is done or make this an optional part of the configuration. >>> >>> This second one is the thing to do…how quickly can we make it happen? >>> >>> >>>>> >>>>> We’re getting a "Too many connections” error when running production…there are only ~100 jobs running. Is this a real limit or just a parameters we can increase (or something else). >>>>> >>>>> Thanks, Matt >>>>> >>>>> >>>>> Fri Dec 26 00:41:28 EST 2014 :: DatabaseConditionsManager :: CONFIG :: host jmysql.jlab.org >>>>> Fri Dec 26 00:41:28 EST 2014 :: DatabaseConditionsManager :: CONFIG :: port 3306 >>>>> Fri Dec 26 00:41:28 EST 2014 :: DatabaseConditionsManager :: CONFIG :: user hpsuser >>>>> Fri Dec 26 00:41:28 EST 2014 :: DatabaseConditionsManager :: CONFIG :: database hps_conditions >>>>> Exception in thread "main" java.lang.RuntimeException: Failed to connect to database: jdbc:mysql://jmysql.jlab.org:3306/ >>>>> at org.hps.conditions.database.ConnectionParameters.createConnection(ConnectionParameters.java:122) >>>>> at org.hps.conditions.database.DatabaseConditionsManager.openConnection(DatabaseConditionsManager.java:150) >>>>> at org.hps.conditions.database.DatabaseConditionsManager.initialize(DatabaseConditionsManager.java:293) >>>>> at org.hps.conditions.database.DatabaseConditionsManager.setDetector(DatabaseConditionsManager.java:240) >>>>> at org.hps.evio.EvioToLcio.run(EvioToLcio.java:272) >>>>> at org.hps.evio.EvioToLcio.main(EvioToLcio.java:99) >>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:525) >>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386) >>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015) >>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989) >>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975) >>>>> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1112) >>>>> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2486) >>>>> at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519) >>>>> at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304) >>>>> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) >>>>> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:525) >>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>>>> at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) >>>>> at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346) >>>>> at java.sql.DriverManager.getConnection(DriverManager.java:579) >>>>> at java.sql.DriverManager.getConnection(DriverManager.java:190) >>>>> at org.hps.conditions.database.ConnectionParameters.createConnection(ConnectionParameters.java:119) >> >> >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the HPS-SOFTWARE list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the HPS-SOFTWARE list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=HPS-SOFTWARE&A=1