Skip to content

Conversation

@kiwiflyer
Copy link
Contributor

After discussion with @miguelaferreira on the previous PR related to Cloudstack-9285, we decided on adding additional exception logging for this issue.

After adding it, the logs look like this in our lab:

2016-04-07 15:44:03,298 WARN cloud.agent.Agent (logid:7225632a) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: Connection closed with -1 on reading size. <<-- new exception logging
2016-04-07 15:44:03,298 INFO cloud.agent.Agent (logid:7225632a) Attempted to connect to the server, but received an unexpected exception, trying again... << --original logging from previous PR.

@GabrielBrascher
Copy link
Member

@kiwiflyer shouldn't the s_logger.warn("NIO Connection Exception " + e); be after the info s_logger.info("Attempted to connect....")?

@kiwiflyer
Copy link
Contributor Author

@GabrielBrascher The reason I placed it before is because the follow on log message is basically informing that a reconnect attempt was going to be attempted next. I'm fine reordering those if that's what you'd prefer.

@GabrielBrascher
Copy link
Member

Got it @kiwiflyer. I am good with the way it is.
Simple logger change, the code LGTM.

@pdube
Copy link
Contributor

pdube commented Apr 8, 2016

Simply added a log warning. Built it LGTM
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CloudStack Developer Tools - Checkstyle Configuration SUCCESS [ 2.360 s]
[INFO] Apache CloudStack ................................. SUCCESS [ 3.182 s]
[INFO] Apache CloudStack Maven Conventions Parent ........ SUCCESS [ 2.313 s]
[INFO] Apache CloudStack Framework - Managed Context ..... SUCCESS [ 7.212 s]
[INFO] Apache CloudStack Utils ........................... SUCCESS [ 30.024 s]
[INFO] Apache CloudStack Framework ....................... SUCCESS [ 0.086 s]
[INFO] Apache CloudStack Framework - Event Notification .. SUCCESS [ 13.763 s]
[INFO] Apache CloudStack Framework - Configuration ....... SUCCESS [ 6.532 s]
[INFO] Apache CloudStack API ............................. SUCCESS [ 29.538 s]
[INFO] Apache CloudStack Framework - REST ................ SUCCESS [ 3.283 s]
[INFO] Apache CloudStack Framework - IPC ................. SUCCESS [ 6.880 s]
[INFO] Apache CloudStack Cloud Engine .................... SUCCESS [ 0.065 s]
[INFO] Apache CloudStack Cloud Engine API ................ SUCCESS [ 7.692 s]
[INFO] Apache CloudStack Framework - Security ............ SUCCESS [ 3.352 s]
[INFO] Apache CloudStack Core ............................ SUCCESS [ 52.743 s]
[INFO] Apache CloudStack Agents .......................... SUCCESS [ 12.096 s]
[INFO] Apache CloudStack Framework - Clustering .......... SUCCESS [ 6.700 s]
[INFO] Apache CloudStack Framework - Event Notification .. SUCCESS [ 1.773 s]
[INFO] Apache CloudStack Cloud Engine Schema Component ... SUCCESS [ 17.664 s]
[INFO] Apache CloudStack Framework - Jobs ................ SUCCESS [ 6.236 s]
[INFO] Apache CloudStack Cloud Engine Internal Components API SUCCESS [ 3.874 s]
[INFO] Apache CloudStack Server .......................... SUCCESS [ 56.665 s]
[INFO] Apache CloudStack Framework - Quota ............... SUCCESS [ 7.043 s]
[INFO] Apache CloudStack Usage Server .................... SUCCESS [ 7.046 s]
[INFO] Apache CloudStack Cloud Engine Orchestration Component SUCCESS [ 16.652 s]
[INFO] Apache CloudStack Cloud Services .................. SUCCESS [ 0.048 s]
[INFO] Apache CloudStack Secondary Storage ............... SUCCESS [ 0.459 s]
[INFO] Apache CloudStack Secondary Storage Service ....... SUCCESS [ 7.624 s]
[INFO] Apache CloudStack Engine Storage Component ........ SUCCESS [ 12.711 s]
[INFO] Apache CloudStack Engine Storage Volume Component . SUCCESS [ 6.630 s]
[INFO] Apache CloudStack Engine Storage Image Component .. SUCCESS [ 3.930 s]
[INFO] Apache CloudStack Engine Storage Data Motion Component SUCCESS [ 3.298 s]
[INFO] Apache CloudStack Engine Storage Cache Component .. SUCCESS [ 3.176 s]
[INFO] Apache CloudStack Engine Storage Snapshot Component SUCCESS [ 10.710 s]
[INFO] Apache CloudStack Cloud Engine API ................ SUCCESS [ 1.993 s]
[INFO] Apache CloudStack Cloud Engine Service ............ SUCCESS [ 6.200 s]
[INFO] Apache CloudStack Plugin POM ...................... SUCCESS [ 0.389 s]
[INFO] Apache CloudStack Plugin - API Rate Limit ......... SUCCESS [ 7.510 s]
[INFO] Apache CloudStack Plugin - Storage Volume default provider SUCCESS [ 2.485 s]
[INFO] Apache CloudStack Plugin - Storage Volume SolidFire Provider SUCCESS [ 5.964 s]
[INFO] Apache CloudStack Plugin - API SolidFire .......... SUCCESS [ 3.038 s]
[INFO] Apache CloudStack Plugin - API Discovery .......... SUCCESS [ 4.980 s]
[INFO] Apache CloudStack Plugin - ACL Static Role Based .. SUCCESS [ 1.830 s]
[INFO] Apache CloudStack Plugin - Host Anti-Affinity Processor SUCCESS [ 2.796 s]
[INFO] Apache CloudStack Plugin - Explicit Dedication Processor SUCCESS [ 2.379 s]
[INFO] Apache CloudStack Plugin - User Concentrated Pod Deployment Planner SUCCESS [ 2.108 s]
[INFO] Apache CloudStack Plugin - User Dispersing Deployment Planner SUCCESS [ 2.554 s]
[INFO] Apache CloudStack Plugin - Implicit Dedication Planner SUCCESS [ 7.570 s]
[INFO] Apache CloudStack Plugin - Skip Heurestics Planner SUCCESS [ 1.979 s]
[INFO] Apache CloudStack Plugin - Host Allocator Random .. SUCCESS [ 2.152 s]
[INFO] Apache CloudStack Plugin - Dedicated Resources .... SUCCESS [ 7.137 s]
[INFO] Apache CloudStack Plugin - Hypervisor OracleVM .... SUCCESS [ 2.642 s]
[INFO] Apache CloudStack Plugin - Open vSwitch ........... SUCCESS [ 3.167 s]
[INFO] Apache CloudStack Plugin - Hypervisor XenServer ... SUCCESS [ 23.396 s]
[INFO] Apache CloudStack Plugin - Hypervisor KVM ......... SUCCESS [ 12.189 s]
[INFO] Apache CloudStack Plugin - RabbitMQ Event Bus ..... SUCCESS [ 2.259 s]
[INFO] Apache CloudStack Plugin - In Memory Event Bus .... SUCCESS [ 3.250 s]
[INFO] Apache CloudStack Plugin - Kafka Event Bus ........ SUCCESS [ 1.599 s]
[INFO] Apache CloudStack Plugin - Hypervisor Baremetal ... SUCCESS [ 3.684 s]
[INFO] Apache CloudStack Plugin - Hypervisor UCS ......... SUCCESS [ 2.491 s]
[INFO] Apache CloudStack Plugin - Hypervisor Hyper-V ..... SUCCESS [ 5.114 s]
[INFO] Apache CloudStack Plugin - Hypervisor OracleVM3 ... SUCCESS [ 21.870 s]
[INFO] Apache CloudStack Plugin - Network Elastic Load Balancer SUCCESS [ 5.239 s]
[INFO] Apache CloudStack Plugin - Network Internal Load Balancer SUCCESS [ 7.841 s]
[INFO] Apache CloudStack Framework - Spring Life Cycle ... SUCCESS [ 2.411 s]
[INFO] Apache CloudStack Plugin - Network Juniper Contrail SUCCESS [ 10.529 s]
[INFO] Apache CloudStack Plugin - Palo Alto .............. SUCCESS [ 5.979 s]
[INFO] Apache CloudStack Plugin - Network Netscaler ...... SUCCESS [ 3.244 s]
[INFO] Apache CloudStack Plugin - Network Nicira NVP ..... SUCCESS [ 14.019 s]
[INFO] Apache CloudStack Plugin - BigSwitch Virtual Network Segment SUCCESS [ 8.554 s]
[INFO] Apache CloudStack Plugin - Network Brocade VCS .... SUCCESS [ 7.312 s]
[INFO] Apache CloudStack Plugin - Midokura Midonet ....... SUCCESS [ 5.131 s]
[INFO] Apache CloudStack Plugin - Stratosphere SSP ....... SUCCESS [ 5.736 s]
[INFO] Apache CloudStack Plugin - Network Opendaylight ... SUCCESS [ 4.878 s]
[INFO] Apache CloudStack Plugin - Storage Allocator Random SUCCESS [ 1.776 s]
[INFO] Apache CloudStack Plugin - User Authenticator LDAP SUCCESS [ 3.555 s]
[INFO] Apache CloudStack Plugin - User Authenticator MD5 . SUCCESS [ 4.386 s]
[INFO] Apache CloudStack Plugin - User Authenticator PBKDF2-SHA-256 SUCCESS [ 6.057 s]
[INFO] Apache CloudStack Plugin - User Authenticator Plain Text SUCCESS [ 1.892 s]
[INFO] Apache CloudStack Plugin - User Authenticator SAML2 SUCCESS [ 27.368 s]
[INFO] Apache CloudStack Plugin - User Authenticator SHA256 Salted SUCCESS [ 4.858 s]
[INFO] Apache CloudStack Plugin - Dns Notifier Example ... SUCCESS [ 1.677 s]
[INFO] Apache CloudStack Plugin - Storage Image S3 provider SUCCESS [ 1.772 s]
[INFO] Apache CloudStack Plugin - Storage Image Swift provider SUCCESS [ 1.784 s]
[INFO] Apache CloudStack Plugin - Storage Image default provider SUCCESS [ 1.824 s]
[INFO] Apache CloudStack Plugin - Storage Image sample provider SUCCESS [ 1.644 s]
[INFO] Apache CloudStack Plugin - Storage Volume Nexenta Provider SUCCESS [ 4.784 s]
[INFO] Apache CloudStack Plugin - Storage Volume CloudByte Provider SUCCESS [ 3.230 s]
[INFO] Apache CloudStack Plugin - Storage Volume sample provider SUCCESS [ 2.021 s]
[INFO] Apache CloudStack Plugin - SNMP Alerts ............ SUCCESS [ 4.050 s]
[INFO] Apache CloudStack Plugin - Syslog Alerts .......... SUCCESS [ 3.824 s]
[INFO] Apache CloudStack Plugin - Network VXLAN .......... SUCCESS [ 4.817 s]
[INFO] Apache CloudStack Plugin - GloboDNS ............... SUCCESS [ 8.192 s]
[INFO] Apache CloudStack Plugin - Quota Service .......... SUCCESS [ 6.589 s]
[INFO] Apache CloudStack Framework - Spring Module ....... SUCCESS [ 6.151 s]
[INFO] Apache CloudStack Secondary Storage Controller .... SUCCESS [ 2.679 s]
[INFO] Apache CloudStack Client UI ....................... SUCCESS [ 11.527 s]
[INFO] Apache CloudStack Console Proxy - RDP Client ...... SUCCESS [ 10.351 s]
[INFO] Apache CloudStack Console Proxy ................... SUCCESS [ 0.330 s]
[INFO] Apache CloudStack Console Proxy - Server .......... SUCCESS [ 4.366 s]
[INFO] Apache CloudStack Framework - QuickCloud .......... SUCCESS [ 0.113 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:54 min
[INFO] Finished at: 2016-04-08T13:55:28-05:00
[INFO] Final Memory: 114M/759M
[INFO] ------------------------------------------------------------------------

@swill
Copy link
Contributor

swill commented Apr 8, 2016

Can you squash your commits and do a push -f to update your branch so this gets merged as a single commit?

This does not change anything that could fail via integration testing, so I don't think we need to run CI against this PR. I have also reviewed this code and it LGTM, so I think this one is ready...

@kiwiflyer kiwiflyer force-pushed the 9285_exception_log branch from 37f8fd1 to f494d10 Compare April 8, 2016 21:38
@kiwiflyer
Copy link
Contributor Author

@swill Squashed and force pushed.

@swill
Copy link
Contributor

swill commented Apr 8, 2016

Thank you @kiwiflyer. Once Jenkins finishes I will merge this since we have the code reviews we need and there is no logical changes that need to be tested (since @pdube has shown the code compiles).

@swill
Copy link
Contributor

swill commented Apr 9, 2016

@kiwiflyer Jenkins is failing. Not sure how the error could be related to your code changes, so can you do a push -f again to kick it off again.

Here is the tests that are failing:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest
Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.702 sec <<< FAILURE! - in org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest
getDefaultNetworkForBasicSGWrongZoneType(org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest)  Time elapsed: 0.567 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<com.cloud.utils.exception.CloudRuntimeException> but was<java.lang.NoSuchMethodError>
    at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest.getDefaultNetworkForBasicSGWrongZoneType(SecondaryStorageManagerTest.java:171)

getDefaultNetworkForAdvancedNonSG(org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest)  Time elapsed: 0.003 sec  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.getDefaultNetworkForAdvancedZone(Lcom/cloud/dc/DataCenter;)Lcom/cloud/network/dao/NetworkVO;
    at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest.getDefaultNetworkForAdvancedNonSG(SecondaryStorageManagerTest.java:79)

getDefaultNetworkForBasicNonSG(org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest)  Time elapsed: 0.002 sec  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.getDefaultNetworkForBasicZone(Lcom/cloud/dc/DataCenter;)Lcom/cloud/network/dao/NetworkVO;
    at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest.getDefaultNetworkForBasicNonSG(SecondaryStorageManagerTest.java:124)

getDefaultNetworkForBasicSG(org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest)  Time elapsed: 0.002 sec  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.getDefaultNetworkForBasicZone(Lcom/cloud/dc/DataCenter;)Lcom/cloud/network/dao/NetworkVO;
    at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest.getDefaultNetworkForBasicSG(SecondaryStorageManagerTest.java:147)

getDefaultNetworkForAdvancedSG(org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest)  Time elapsed: 0.002 sec  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.getDefaultNetworkForAdvancedZone(Lcom/cloud/dc/DataCenter;)Lcom/cloud/network/dao/NetworkVO;
    at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest.getDefaultNetworkForAdvancedSG(SecondaryStorageManagerTest.java:102)

getDefaultNetworkForAdvancedWrongZoneType(org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest)  Time elapsed: 0.002 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<com.cloud.utils.exception.CloudRuntimeException> but was<java.lang.NoSuchMethodError>
    at org.apache.cloudstack.secondarystorage.SecondaryStorageManagerTest.getDefaultNetworkForAdvancedWrongZoneType(SecondaryStorageManagerTest.java:190)


Results :

Tests in error: 
  SecondaryStorageManagerTest.getDefaultNetworkForAdvancedNonSG:79 NoSuchMethod ...
  SecondaryStorageManagerTest.getDefaultNetworkForAdvancedSG:102 NoSuchMethod or...
  SecondaryStorageManagerTest.getDefaultNetworkForAdvancedWrongZoneType ?  Unexp...
  SecondaryStorageManagerTest.getDefaultNetworkForBasicNonSG:124 NoSuchMethod or...
  SecondaryStorageManagerTest.getDefaultNetworkForBasicSG:147 NoSuchMethod org.a...
  SecondaryStorageManagerTest.getDefaultNetworkForBasicSGWrongZoneType ?  Unexpe...

Tests run: 6, Failures: 0, Errors: 6, Skipped: 0

@miguelaferreira
Copy link
Contributor

@kiwiflyer I can't seem to find that PR discussion you mention. Maybe you got the wrong handle?

@kiwiflyer
Copy link
Contributor Author

I apologize @miguelaferreira. You are correct, I tagged the wrong handle. It was actually @rafaelweingartner.

@swill
Copy link
Contributor

swill commented Apr 11, 2016

@kiwiflyer Sorry to do this to you, but would you mind squashing your commits and doing a push -f again? Thanks...

@kiwiflyer kiwiflyer force-pushed the 9285_exception_log branch from 9b8dccc to c729316 Compare April 11, 2016 13:29
@kiwiflyer
Copy link
Contributor Author

Squashed.

@kiwiflyer kiwiflyer force-pushed the 9285_exception_log branch from c729316 to 698107d Compare April 11, 2016 14:44
@kiwiflyer kiwiflyer force-pushed the 9285_exception_log branch from 698107d to 5f062f1 Compare April 11, 2016 22:06
@kiwiflyer
Copy link
Contributor Author

@swill - I force pushed it again and Jenkins is finally passing.

@swill
Copy link
Contributor

swill commented Apr 12, 2016

@kiwiflyer woohoo!!! thank you for all the support on this. i will merge this in the morning with my next batch.

@asfgit asfgit merged commit 5f062f1 into apache:4.7 Apr 12, 2016
asfgit pushed a commit that referenced this pull request Apr 12, 2016
Cloudstack-9285 exception log additionAfter discussion with @miguelaferreira on the previous PR related to Cloudstack-9285, we decided on adding additional exception logging for this issue.

After adding it, the logs look like this in our lab:

2016-04-07 15:44:03,298 WARN  [cloud.agent.Agent] (Agent-Handler-1:null) (logid:7225632a) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: Connection closed with -1 on reading size.  <<-- new exception logging
2016-04-07 15:44:03,298 INFO  [cloud.agent.Agent] (Agent-Handler-1:null) (logid:7225632a) Attempted to connect to the server, but received an unexpected exception, trying again... << --original logging from previous PR.

* pr/1479:
  Additional exception logging for Cloudstack-9285

Signed-off-by: Will Stevens <williamstevens@gmail.com>
try {
_connection.start();
} catch (final NioConnectionException e) {
s_logger.warn("NIO Connection Exception " + e);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it is unlikely that these two lines will appear in the log sequentially, why aren't these two log messages combined into one WARN message?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants