Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
260.36641ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:33:52 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 260.505551ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.....X.....] duration:
2.560026976s trace:
https://jaeger.dev.storj.io/trace/a5ae6135be9170
FAIL (2025-12-23 15:33:50 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.560050258s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.................................X..............X...........] duration:
812.942067ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:33:25 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 812.97189ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..............................X......................X......] duration:
2.521530718s trace:
https://jaeger.dev.storj.io/trace/20ee6d11665285b1
FAIL (2025-12-23 15:33:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450238296 (432h0m51.27794669s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450284693 (432h0m51.277993094s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450928506 (432h0m51.2786369s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451455886 (432h0m51.279164277s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451726821 (432h0m51.279435218s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452155821 (432h0m51.279864216s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452617301 (432h0m51.280325695s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453031906 (432h0m51.280740306s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453464313 (432h0m51.281172704s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453833854 (432h0m51.281542232s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.873749393 (432h0m51.701457785s) [upload=449474]: Next returned nil segment
Dec 23 15:28:50.873762459 (432h0m51.701470849s) [upload=449474]: Waiting for error group managing segments...
Dec 23 15:28:51.042505138 (432h0m51.870213541s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.047157036 (432h0m51.874865439s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.061283412 (432h0m51.888991827s) [noise=true , address=185.167.204.206:4066 , piece_id=TXAR7UAXS4MKKGJHLSTE7KW53JKPV6BZVMUXMTPATMYM3SLKOGGQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.062400429 (432h0m51.89010882s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.066410014 (432h0m51.894118413s) [noise=true , address=185.167.204.205:4009 , piece_id=QOWW2X6YAMSGYW3BC7ZRPNLZ6ZZJ3CQVSYACJDWHLJNAMDMBQPBA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.095270986 (432h0m51.92297939s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.100596666 (432h0m51.928305061s) [noise=true , address=185.167.204.205:4019 , piece_id=JAQWQNVVW7DUV5H7O52ILYHN55VVWADGGAYFBDK4VM7AZWNAXCRQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.103560142 (432h0m51.93126853s) [noise=true , address=81.171.14.70:4025 , piece_id=WGTWZVCDY4N2FGSOCKQEDS5MC25AHPI2MFBF2VRO47ZF5E65CY2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.118031585 (432h0m51.945739979s) [noise=true , address=185.167.204.205:4021 , piece_id=7D7PQWUUQ6OV66VP73XGLAURNLQFOX5YT3MURRQSAN66VKA3DKYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.128062622 (432h0m51.955771018s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.134397721 (432h0m51.962106124s) [noise=true , address=89.149.201.147:4009 , piece_id=NLGLDZI6VRLXIP2S3G6E6BCK6ERGF3LKIHGKKYDRTDU7IAE3MOEA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.163106517 (432h0m51.990814919s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.187215887 (432h0m52.014924285s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.191385548 (432h0m52.019093943s) [noise=true , address=185.167.204.205:4025 , piece_id=STHOVH3EE4E5EGHV3YXD7PAK7HL64A4YZX5I2L7REVBDM7SOXG4A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.232740501 (432h0m52.060448892s) [noise=true , address=185.167.204.206:4067 , piece_id=A5Z7XL6JY37OWKH3ZDC5A2W73ERAQCS2WIOXASUHEX3CPYCG2PYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.254566742 (432h0m52.082275133s) [noise=true , address=185.167.204.206:4052 , piece_id=KWPFK2SOTESIA4YWLNPM6CES7DUPWCSVAI3E5UDYK6YSNM3EFEOA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.292399281 (432h0m52.120107676s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.495257157 (432h0m52.322965538s) [noise=true , address=185.167.204.205:4018 , piece_id=XAIHJLWSUIGGHLKETNCWUAUCA76QOHNN3BD6H3H47XQMONNHDVCQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.509483881 (432h0m52.337192272s) [noise=true , address=81.171.14.70:4035 , piece_id=YIAIQHW5HLQS63JX6TITC3YZKBNH3BC5YYJ7DNC7EWCXL6ILDHOQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.519317972 (432h0m52.347026369s) [noise=true , address=81.171.14.70:4033 , piece_id=4RARLBHMGEOWN6SS2DKQ6DVWSUO54G4LV5UP2LYOGJAZDEEWD2CA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.540953811 (432h0m52.368662212s) [noise=true , address=89.149.201.147:4017 , piece_id=YKTOFL6L2A2OSVXSM2NW5MGXFHVXH4QYBIO2XWLWOH5ANRM2NHYA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.555541803 (432h0m52.383250199s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.561754672 (432h0m52.389463063s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.565484488 (432h0m52.393192887s) [noise=true , address=89.149.201.147:4008 , piece_id=2FME7O25WWJGPW3QR7SNATQXK4UA2KZIN7J2ZUNRQTM67NVR2Y2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.580597563 (432h0m52.408305955s) [noise=true , address=81.171.14.70:4027 , piece_id=FRRLBX6EMKUAEFNELKYS3WPNG6CDYX7WCBLVPHAW3AZGQBMB3V3A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596579728 (432h0m52.424288127s) [noise=true , address=81.171.14.70:4043 , piece_id=MJM5262FY7HLPJHDZU7Z5FOZKGQG6XLIEGHB3W3PECXJOUATGNBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596646083 (432h0m52.424354476s) [seg_pos={0 0}, upload=449474]: Segment reached optimal threshold of 26 pieces.
Dec 23 15:28:51.597463826 (432h0m52.425172227s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.597488049 (432h0m52.425196452s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598037697 (432h0m52.425746098s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.598080201 (432h0m52.425788602s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598169558 (432h0m52.425877955s) [seg_pos={0 0}, upload=449474]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 23 15:28:51.598361733 (432h0m52.426070137s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 15:28:51.598386842 (432h0m52.426095234s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 15:29:47.549935238 (432h1m48.377643635s) [upload=449474]: Done uploading segments...
Dec 23 15:29:47.550073441 (432h1m48.377781841s) [upload=449474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.521636074s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
261.632989ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:32:47 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 261.655436ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [...............................X......................X.....] duration:
2.512856187s trace:
https://jaeger.dev.storj.io/trace/20ee6d11665285b1
FAIL (2025-12-23 15:32:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450238296 (432h0m51.27794669s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450284693 (432h0m51.277993094s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450928506 (432h0m51.2786369s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451455886 (432h0m51.279164277s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451726821 (432h0m51.279435218s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452155821 (432h0m51.279864216s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452617301 (432h0m51.280325695s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453031906 (432h0m51.280740306s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453464313 (432h0m51.281172704s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453833854 (432h0m51.281542232s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.873749393 (432h0m51.701457785s) [upload=449474]: Next returned nil segment
Dec 23 15:28:50.873762459 (432h0m51.701470849s) [upload=449474]: Waiting for error group managing segments...
Dec 23 15:28:51.042505138 (432h0m51.870213541s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.047157036 (432h0m51.874865439s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.061283412 (432h0m51.888991827s) [noise=true , address=185.167.204.206:4066 , piece_id=TXAR7UAXS4MKKGJHLSTE7KW53JKPV6BZVMUXMTPATMYM3SLKOGGQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.062400429 (432h0m51.89010882s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.066410014 (432h0m51.894118413s) [noise=true , address=185.167.204.205:4009 , piece_id=QOWW2X6YAMSGYW3BC7ZRPNLZ6ZZJ3CQVSYACJDWHLJNAMDMBQPBA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.095270986 (432h0m51.92297939s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.100596666 (432h0m51.928305061s) [noise=true , address=185.167.204.205:4019 , piece_id=JAQWQNVVW7DUV5H7O52ILYHN55VVWADGGAYFBDK4VM7AZWNAXCRQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.103560142 (432h0m51.93126853s) [noise=true , address=81.171.14.70:4025 , piece_id=WGTWZVCDY4N2FGSOCKQEDS5MC25AHPI2MFBF2VRO47ZF5E65CY2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.118031585 (432h0m51.945739979s) [noise=true , address=185.167.204.205:4021 , piece_id=7D7PQWUUQ6OV66VP73XGLAURNLQFOX5YT3MURRQSAN66VKA3DKYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.128062622 (432h0m51.955771018s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.134397721 (432h0m51.962106124s) [noise=true , address=89.149.201.147:4009 , piece_id=NLGLDZI6VRLXIP2S3G6E6BCK6ERGF3LKIHGKKYDRTDU7IAE3MOEA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.163106517 (432h0m51.990814919s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.187215887 (432h0m52.014924285s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.191385548 (432h0m52.019093943s) [noise=true , address=185.167.204.205:4025 , piece_id=STHOVH3EE4E5EGHV3YXD7PAK7HL64A4YZX5I2L7REVBDM7SOXG4A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.232740501 (432h0m52.060448892s) [noise=true , address=185.167.204.206:4067 , piece_id=A5Z7XL6JY37OWKH3ZDC5A2W73ERAQCS2WIOXASUHEX3CPYCG2PYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.254566742 (432h0m52.082275133s) [noise=true , address=185.167.204.206:4052 , piece_id=KWPFK2SOTESIA4YWLNPM6CES7DUPWCSVAI3E5UDYK6YSNM3EFEOA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.292399281 (432h0m52.120107676s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.495257157 (432h0m52.322965538s) [noise=true , address=185.167.204.205:4018 , piece_id=XAIHJLWSUIGGHLKETNCWUAUCA76QOHNN3BD6H3H47XQMONNHDVCQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.509483881 (432h0m52.337192272s) [noise=true , address=81.171.14.70:4035 , piece_id=YIAIQHW5HLQS63JX6TITC3YZKBNH3BC5YYJ7DNC7EWCXL6ILDHOQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.519317972 (432h0m52.347026369s) [noise=true , address=81.171.14.70:4033 , piece_id=4RARLBHMGEOWN6SS2DKQ6DVWSUO54G4LV5UP2LYOGJAZDEEWD2CA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.540953811 (432h0m52.368662212s) [noise=true , address=89.149.201.147:4017 , piece_id=YKTOFL6L2A2OSVXSM2NW5MGXFHVXH4QYBIO2XWLWOH5ANRM2NHYA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.555541803 (432h0m52.383250199s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.561754672 (432h0m52.389463063s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.565484488 (432h0m52.393192887s) [noise=true , address=89.149.201.147:4008 , piece_id=2FME7O25WWJGPW3QR7SNATQXK4UA2KZIN7J2ZUNRQTM67NVR2Y2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.580597563 (432h0m52.408305955s) [noise=true , address=81.171.14.70:4027 , piece_id=FRRLBX6EMKUAEFNELKYS3WPNG6CDYX7WCBLVPHAW3AZGQBMB3V3A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596579728 (432h0m52.424288127s) [noise=true , address=81.171.14.70:4043 , piece_id=MJM5262FY7HLPJHDZU7Z5FOZKGQG6XLIEGHB3W3PECXJOUATGNBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596646083 (432h0m52.424354476s) [seg_pos={0 0}, upload=449474]: Segment reached optimal threshold of 26 pieces.
Dec 23 15:28:51.597463826 (432h0m52.425172227s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.597488049 (432h0m52.425196452s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598037697 (432h0m52.425746098s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.598080201 (432h0m52.425788602s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598169558 (432h0m52.425877955s) [seg_pos={0 0}, upload=449474]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 23 15:28:51.598361733 (432h0m52.426070137s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 15:28:51.598386842 (432h0m52.426095234s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 15:29:47.549935238 (432h1m48.377643635s) [upload=449474]: Done uploading segments...
Dec 23 15:29:47.550073441 (432h1m48.377781841s) [upload=449474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.512880622s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.....X....] duration:
2.661582341s trace:
https://jaeger.dev.storj.io/trace/a5ae6135be9170
FAIL (2025-12-23 15:32:42 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.661606466s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..................................X..............X..........] duration:
915.951011ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:32:30 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 915.974092ms
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
255.516714ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:32:02 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 255.551993ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [................................X......................X....] duration:
2.488052809s trace:
https://jaeger.dev.storj.io/trace/20ee6d11665285b1
FAIL (2025-12-23 15:31:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450238296 (432h0m51.27794669s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450284693 (432h0m51.277993094s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450928506 (432h0m51.2786369s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451455886 (432h0m51.279164277s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451726821 (432h0m51.279435218s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452155821 (432h0m51.279864216s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452617301 (432h0m51.280325695s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453031906 (432h0m51.280740306s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453464313 (432h0m51.281172704s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453833854 (432h0m51.281542232s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.873749393 (432h0m51.701457785s) [upload=449474]: Next returned nil segment
Dec 23 15:28:50.873762459 (432h0m51.701470849s) [upload=449474]: Waiting for error group managing segments...
Dec 23 15:28:51.042505138 (432h0m51.870213541s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.047157036 (432h0m51.874865439s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.061283412 (432h0m51.888991827s) [noise=true , address=185.167.204.206:4066 , piece_id=TXAR7UAXS4MKKGJHLSTE7KW53JKPV6BZVMUXMTPATMYM3SLKOGGQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.062400429 (432h0m51.89010882s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.066410014 (432h0m51.894118413s) [noise=true , address=185.167.204.205:4009 , piece_id=QOWW2X6YAMSGYW3BC7ZRPNLZ6ZZJ3CQVSYACJDWHLJNAMDMBQPBA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.095270986 (432h0m51.92297939s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.100596666 (432h0m51.928305061s) [noise=true , address=185.167.204.205:4019 , piece_id=JAQWQNVVW7DUV5H7O52ILYHN55VVWADGGAYFBDK4VM7AZWNAXCRQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.103560142 (432h0m51.93126853s) [noise=true , address=81.171.14.70:4025 , piece_id=WGTWZVCDY4N2FGSOCKQEDS5MC25AHPI2MFBF2VRO47ZF5E65CY2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.118031585 (432h0m51.945739979s) [noise=true , address=185.167.204.205:4021 , piece_id=7D7PQWUUQ6OV66VP73XGLAURNLQFOX5YT3MURRQSAN66VKA3DKYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.128062622 (432h0m51.955771018s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.134397721 (432h0m51.962106124s) [noise=true , address=89.149.201.147:4009 , piece_id=NLGLDZI6VRLXIP2S3G6E6BCK6ERGF3LKIHGKKYDRTDU7IAE3MOEA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.163106517 (432h0m51.990814919s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.187215887 (432h0m52.014924285s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.191385548 (432h0m52.019093943s) [noise=true , address=185.167.204.205:4025 , piece_id=STHOVH3EE4E5EGHV3YXD7PAK7HL64A4YZX5I2L7REVBDM7SOXG4A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.232740501 (432h0m52.060448892s) [noise=true , address=185.167.204.206:4067 , piece_id=A5Z7XL6JY37OWKH3ZDC5A2W73ERAQCS2WIOXASUHEX3CPYCG2PYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.254566742 (432h0m52.082275133s) [noise=true , address=185.167.204.206:4052 , piece_id=KWPFK2SOTESIA4YWLNPM6CES7DUPWCSVAI3E5UDYK6YSNM3EFEOA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.292399281 (432h0m52.120107676s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.495257157 (432h0m52.322965538s) [noise=true , address=185.167.204.205:4018 , piece_id=XAIHJLWSUIGGHLKETNCWUAUCA76QOHNN3BD6H3H47XQMONNHDVCQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.509483881 (432h0m52.337192272s) [noise=true , address=81.171.14.70:4035 , piece_id=YIAIQHW5HLQS63JX6TITC3YZKBNH3BC5YYJ7DNC7EWCXL6ILDHOQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.519317972 (432h0m52.347026369s) [noise=true , address=81.171.14.70:4033 , piece_id=4RARLBHMGEOWN6SS2DKQ6DVWSUO54G4LV5UP2LYOGJAZDEEWD2CA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.540953811 (432h0m52.368662212s) [noise=true , address=89.149.201.147:4017 , piece_id=YKTOFL6L2A2OSVXSM2NW5MGXFHVXH4QYBIO2XWLWOH5ANRM2NHYA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.555541803 (432h0m52.383250199s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.561754672 (432h0m52.389463063s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.565484488 (432h0m52.393192887s) [noise=true , address=89.149.201.147:4008 , piece_id=2FME7O25WWJGPW3QR7SNATQXK4UA2KZIN7J2ZUNRQTM67NVR2Y2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.580597563 (432h0m52.408305955s) [noise=true , address=81.171.14.70:4027 , piece_id=FRRLBX6EMKUAEFNELKYS3WPNG6CDYX7WCBLVPHAW3AZGQBMB3V3A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596579728 (432h0m52.424288127s) [noise=true , address=81.171.14.70:4043 , piece_id=MJM5262FY7HLPJHDZU7Z5FOZKGQG6XLIEGHB3W3PECXJOUATGNBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596646083 (432h0m52.424354476s) [seg_pos={0 0}, upload=449474]: Segment reached optimal threshold of 26 pieces.
Dec 23 15:28:51.597463826 (432h0m52.425172227s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.597488049 (432h0m52.425196452s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598037697 (432h0m52.425746098s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.598080201 (432h0m52.425788602s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598169558 (432h0m52.425877955s) [seg_pos={0 0}, upload=449474]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 23 15:28:51.598361733 (432h0m52.426070137s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 15:28:51.598386842 (432h0m52.426095234s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 15:29:47.549935238 (432h1m48.377643635s) [upload=449474]: Done uploading segments...
Dec 23 15:29:47.550073441 (432h1m48.377781841s) [upload=449474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.488093008s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration:
2.674169817s trace:
https://jaeger.dev.storj.io/trace/a5ae6135be9170
FAIL (2025-12-23 15:31:52 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.674196117s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.................................X......................X...] duration:
2.633822412s trace:
https://jaeger.dev.storj.io/trace/20ee6d11665285b1
FAIL (2025-12-23 15:31:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450238296 (432h0m51.27794669s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450284693 (432h0m51.277993094s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450928506 (432h0m51.2786369s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451455886 (432h0m51.279164277s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451726821 (432h0m51.279435218s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452155821 (432h0m51.279864216s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452617301 (432h0m51.280325695s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453031906 (432h0m51.280740306s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453464313 (432h0m51.281172704s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453833854 (432h0m51.281542232s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.873749393 (432h0m51.701457785s) [upload=449474]: Next returned nil segment
Dec 23 15:28:50.873762459 (432h0m51.701470849s) [upload=449474]: Waiting for error group managing segments...
Dec 23 15:28:51.042505138 (432h0m51.870213541s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.047157036 (432h0m51.874865439s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.061283412 (432h0m51.888991827s) [noise=true , address=185.167.204.206:4066 , piece_id=TXAR7UAXS4MKKGJHLSTE7KW53JKPV6BZVMUXMTPATMYM3SLKOGGQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.062400429 (432h0m51.89010882s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.066410014 (432h0m51.894118413s) [noise=true , address=185.167.204.205:4009 , piece_id=QOWW2X6YAMSGYW3BC7ZRPNLZ6ZZJ3CQVSYACJDWHLJNAMDMBQPBA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.095270986 (432h0m51.92297939s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.100596666 (432h0m51.928305061s) [noise=true , address=185.167.204.205:4019 , piece_id=JAQWQNVVW7DUV5H7O52ILYHN55VVWADGGAYFBDK4VM7AZWNAXCRQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.103560142 (432h0m51.93126853s) [noise=true , address=81.171.14.70:4025 , piece_id=WGTWZVCDY4N2FGSOCKQEDS5MC25AHPI2MFBF2VRO47ZF5E65CY2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.118031585 (432h0m51.945739979s) [noise=true , address=185.167.204.205:4021 , piece_id=7D7PQWUUQ6OV66VP73XGLAURNLQFOX5YT3MURRQSAN66VKA3DKYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.128062622 (432h0m51.955771018s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.134397721 (432h0m51.962106124s) [noise=true , address=89.149.201.147:4009 , piece_id=NLGLDZI6VRLXIP2S3G6E6BCK6ERGF3LKIHGKKYDRTDU7IAE3MOEA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.163106517 (432h0m51.990814919s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.187215887 (432h0m52.014924285s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.191385548 (432h0m52.019093943s) [noise=true , address=185.167.204.205:4025 , piece_id=STHOVH3EE4E5EGHV3YXD7PAK7HL64A4YZX5I2L7REVBDM7SOXG4A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.232740501 (432h0m52.060448892s) [noise=true , address=185.167.204.206:4067 , piece_id=A5Z7XL6JY37OWKH3ZDC5A2W73ERAQCS2WIOXASUHEX3CPYCG2PYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.254566742 (432h0m52.082275133s) [noise=true , address=185.167.204.206:4052 , piece_id=KWPFK2SOTESIA4YWLNPM6CES7DUPWCSVAI3E5UDYK6YSNM3EFEOA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.292399281 (432h0m52.120107676s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.495257157 (432h0m52.322965538s) [noise=true , address=185.167.204.205:4018 , piece_id=XAIHJLWSUIGGHLKETNCWUAUCA76QOHNN3BD6H3H47XQMONNHDVCQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.509483881 (432h0m52.337192272s) [noise=true , address=81.171.14.70:4035 , piece_id=YIAIQHW5HLQS63JX6TITC3YZKBNH3BC5YYJ7DNC7EWCXL6ILDHOQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.519317972 (432h0m52.347026369s) [noise=true , address=81.171.14.70:4033 , piece_id=4RARLBHMGEOWN6SS2DKQ6DVWSUO54G4LV5UP2LYOGJAZDEEWD2CA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.540953811 (432h0m52.368662212s) [noise=true , address=89.149.201.147:4017 , piece_id=YKTOFL6L2A2OSVXSM2NW5MGXFHVXH4QYBIO2XWLWOH5ANRM2NHYA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.555541803 (432h0m52.383250199s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.561754672 (432h0m52.389463063s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.565484488 (432h0m52.393192887s) [noise=true , address=89.149.201.147:4008 , piece_id=2FME7O25WWJGPW3QR7SNATQXK4UA2KZIN7J2ZUNRQTM67NVR2Y2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.580597563 (432h0m52.408305955s) [noise=true , address=81.171.14.70:4027 , piece_id=FRRLBX6EMKUAEFNELKYS3WPNG6CDYX7WCBLVPHAW3AZGQBMB3V3A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596579728 (432h0m52.424288127s) [noise=true , address=81.171.14.70:4043 , piece_id=MJM5262FY7HLPJHDZU7Z5FOZKGQG6XLIEGHB3W3PECXJOUATGNBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596646083 (432h0m52.424354476s) [seg_pos={0 0}, upload=449474]: Segment reached optimal threshold of 26 pieces.
Dec 23 15:28:51.597463826 (432h0m52.425172227s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.597488049 (432h0m52.425196452s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598037697 (432h0m52.425746098s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.598080201 (432h0m52.425788602s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598169558 (432h0m52.425877955s) [seg_pos={0 0}, upload=449474]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 23 15:28:51.598361733 (432h0m52.426070137s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 15:28:51.598386842 (432h0m52.426095234s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 15:29:47.549935238 (432h1m48.377643635s) [upload=449474]: Done uploading segments...
Dec 23 15:29:47.550073441 (432h1m48.377781841s) [upload=449474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.633849881s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...................................X..............X.........] duration:
796.180267ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:31:24 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 796.21138ms
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
260.554889ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:31:09 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 260.590416ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [....................................X..............X........] duration:
804.817164ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:31:05 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 804.844469ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration:
2.53329222s trace:
https://jaeger.dev.storj.io/trace/a5ae6135be9170
FAIL (2025-12-23 15:30:55 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.53332017s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..................................X......................X..] duration:
2.061323238s trace:
https://jaeger.dev.storj.io/trace/20ee6d11665285b1
FAIL (2025-12-23 15:30:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450238296 (432h0m51.27794669s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450284693 (432h0m51.277993094s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450928506 (432h0m51.2786369s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451455886 (432h0m51.279164277s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451726821 (432h0m51.279435218s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452155821 (432h0m51.279864216s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452617301 (432h0m51.280325695s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453031906 (432h0m51.280740306s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453464313 (432h0m51.281172704s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453833854 (432h0m51.281542232s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.873749393 (432h0m51.701457785s) [upload=449474]: Next returned nil segment
Dec 23 15:28:50.873762459 (432h0m51.701470849s) [upload=449474]: Waiting for error group managing segments...
Dec 23 15:28:51.042505138 (432h0m51.870213541s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.047157036 (432h0m51.874865439s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.061283412 (432h0m51.888991827s) [noise=true , address=185.167.204.206:4066 , piece_id=TXAR7UAXS4MKKGJHLSTE7KW53JKPV6BZVMUXMTPATMYM3SLKOGGQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.062400429 (432h0m51.89010882s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.066410014 (432h0m51.894118413s) [noise=true , address=185.167.204.205:4009 , piece_id=QOWW2X6YAMSGYW3BC7ZRPNLZ6ZZJ3CQVSYACJDWHLJNAMDMBQPBA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.095270986 (432h0m51.92297939s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.100596666 (432h0m51.928305061s) [noise=true , address=185.167.204.205:4019 , piece_id=JAQWQNVVW7DUV5H7O52ILYHN55VVWADGGAYFBDK4VM7AZWNAXCRQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.103560142 (432h0m51.93126853s) [noise=true , address=81.171.14.70:4025 , piece_id=WGTWZVCDY4N2FGSOCKQEDS5MC25AHPI2MFBF2VRO47ZF5E65CY2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.118031585 (432h0m51.945739979s) [noise=true , address=185.167.204.205:4021 , piece_id=7D7PQWUUQ6OV66VP73XGLAURNLQFOX5YT3MURRQSAN66VKA3DKYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.128062622 (432h0m51.955771018s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.134397721 (432h0m51.962106124s) [noise=true , address=89.149.201.147:4009 , piece_id=NLGLDZI6VRLXIP2S3G6E6BCK6ERGF3LKIHGKKYDRTDU7IAE3MOEA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.163106517 (432h0m51.990814919s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.187215887 (432h0m52.014924285s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.191385548 (432h0m52.019093943s) [noise=true , address=185.167.204.205:4025 , piece_id=STHOVH3EE4E5EGHV3YXD7PAK7HL64A4YZX5I2L7REVBDM7SOXG4A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.232740501 (432h0m52.060448892s) [noise=true , address=185.167.204.206:4067 , piece_id=A5Z7XL6JY37OWKH3ZDC5A2W73ERAQCS2WIOXASUHEX3CPYCG2PYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.254566742 (432h0m52.082275133s) [noise=true , address=185.167.204.206:4052 , piece_id=KWPFK2SOTESIA4YWLNPM6CES7DUPWCSVAI3E5UDYK6YSNM3EFEOA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.292399281 (432h0m52.120107676s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.495257157 (432h0m52.322965538s) [noise=true , address=185.167.204.205:4018 , piece_id=XAIHJLWSUIGGHLKETNCWUAUCA76QOHNN3BD6H3H47XQMONNHDVCQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.509483881 (432h0m52.337192272s) [noise=true , address=81.171.14.70:4035 , piece_id=YIAIQHW5HLQS63JX6TITC3YZKBNH3BC5YYJ7DNC7EWCXL6ILDHOQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.519317972 (432h0m52.347026369s) [noise=true , address=81.171.14.70:4033 , piece_id=4RARLBHMGEOWN6SS2DKQ6DVWSUO54G4LV5UP2LYOGJAZDEEWD2CA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.540953811 (432h0m52.368662212s) [noise=true , address=89.149.201.147:4017 , piece_id=YKTOFL6L2A2OSVXSM2NW5MGXFHVXH4QYBIO2XWLWOH5ANRM2NHYA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.555541803 (432h0m52.383250199s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.561754672 (432h0m52.389463063s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.565484488 (432h0m52.393192887s) [noise=true , address=89.149.201.147:4008 , piece_id=2FME7O25WWJGPW3QR7SNATQXK4UA2KZIN7J2ZUNRQTM67NVR2Y2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.580597563 (432h0m52.408305955s) [noise=true , address=81.171.14.70:4027 , piece_id=FRRLBX6EMKUAEFNELKYS3WPNG6CDYX7WCBLVPHAW3AZGQBMB3V3A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596579728 (432h0m52.424288127s) [noise=true , address=81.171.14.70:4043 , piece_id=MJM5262FY7HLPJHDZU7Z5FOZKGQG6XLIEGHB3W3PECXJOUATGNBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596646083 (432h0m52.424354476s) [seg_pos={0 0}, upload=449474]: Segment reached optimal threshold of 26 pieces.
Dec 23 15:28:51.597463826 (432h0m52.425172227s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.597488049 (432h0m52.425196452s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598037697 (432h0m52.425746098s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.598080201 (432h0m52.425788602s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598169558 (432h0m52.425877955s) [seg_pos={0 0}, upload=449474]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 23 15:28:51.598361733 (432h0m52.426070137s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 15:28:51.598386842 (432h0m52.426095234s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 15:29:47.549935238 (432h1m48.377643635s) [upload=449474]: Done uploading segments...
Dec 23 15:29:47.550073441 (432h1m48.377781841s) [upload=449474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.061356313s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X..............X.......] duration:
821.63364ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:30:19 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 821.66954ms
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
262.236912ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:30:13 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 262.264967ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................................X..............X......] duration:
821.392204ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:29:59 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 821.416176ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration:
2.472283123s trace:
https://jaeger.dev.storj.io/trace/a5ae6135be9170
FAIL (2025-12-23 15:29:53 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 2.472314185s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [...................................X......................X.] duration:
2.050949363s trace:
https://jaeger.dev.storj.io/trace/20ee6d11665285b1
FAIL (2025-12-23 15:29:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450238296 (432h0m51.27794669s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450284693 (432h0m51.277993094s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450928506 (432h0m51.2786369s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451455886 (432h0m51.279164277s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451726821 (432h0m51.279435218s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452155821 (432h0m51.279864216s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452617301 (432h0m51.280325695s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453031906 (432h0m51.280740306s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453464313 (432h0m51.281172704s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453833854 (432h0m51.281542232s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.873749393 (432h0m51.701457785s) [upload=449474]: Next returned nil segment
Dec 23 15:28:50.873762459 (432h0m51.701470849s) [upload=449474]: Waiting for error group managing segments...
Dec 23 15:28:51.042505138 (432h0m51.870213541s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.047157036 (432h0m51.874865439s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.061283412 (432h0m51.888991827s) [noise=true , address=185.167.204.206:4066 , piece_id=TXAR7UAXS4MKKGJHLSTE7KW53JKPV6BZVMUXMTPATMYM3SLKOGGQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.062400429 (432h0m51.89010882s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.066410014 (432h0m51.894118413s) [noise=true , address=185.167.204.205:4009 , piece_id=QOWW2X6YAMSGYW3BC7ZRPNLZ6ZZJ3CQVSYACJDWHLJNAMDMBQPBA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.095270986 (432h0m51.92297939s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.100596666 (432h0m51.928305061s) [noise=true , address=185.167.204.205:4019 , piece_id=JAQWQNVVW7DUV5H7O52ILYHN55VVWADGGAYFBDK4VM7AZWNAXCRQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.103560142 (432h0m51.93126853s) [noise=true , address=81.171.14.70:4025 , piece_id=WGTWZVCDY4N2FGSOCKQEDS5MC25AHPI2MFBF2VRO47ZF5E65CY2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.118031585 (432h0m51.945739979s) [noise=true , address=185.167.204.205:4021 , piece_id=7D7PQWUUQ6OV66VP73XGLAURNLQFOX5YT3MURRQSAN66VKA3DKYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.128062622 (432h0m51.955771018s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.134397721 (432h0m51.962106124s) [noise=true , address=89.149.201.147:4009 , piece_id=NLGLDZI6VRLXIP2S3G6E6BCK6ERGF3LKIHGKKYDRTDU7IAE3MOEA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.163106517 (432h0m51.990814919s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.187215887 (432h0m52.014924285s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.191385548 (432h0m52.019093943s) [noise=true , address=185.167.204.205:4025 , piece_id=STHOVH3EE4E5EGHV3YXD7PAK7HL64A4YZX5I2L7REVBDM7SOXG4A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.232740501 (432h0m52.060448892s) [noise=true , address=185.167.204.206:4067 , piece_id=A5Z7XL6JY37OWKH3ZDC5A2W73ERAQCS2WIOXASUHEX3CPYCG2PYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.254566742 (432h0m52.082275133s) [noise=true , address=185.167.204.206:4052 , piece_id=KWPFK2SOTESIA4YWLNPM6CES7DUPWCSVAI3E5UDYK6YSNM3EFEOA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.292399281 (432h0m52.120107676s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.495257157 (432h0m52.322965538s) [noise=true , address=185.167.204.205:4018 , piece_id=XAIHJLWSUIGGHLKETNCWUAUCA76QOHNN3BD6H3H47XQMONNHDVCQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.509483881 (432h0m52.337192272s) [noise=true , address=81.171.14.70:4035 , piece_id=YIAIQHW5HLQS63JX6TITC3YZKBNH3BC5YYJ7DNC7EWCXL6ILDHOQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.519317972 (432h0m52.347026369s) [noise=true , address=81.171.14.70:4033 , piece_id=4RARLBHMGEOWN6SS2DKQ6DVWSUO54G4LV5UP2LYOGJAZDEEWD2CA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.540953811 (432h0m52.368662212s) [noise=true , address=89.149.201.147:4017 , piece_id=YKTOFL6L2A2OSVXSM2NW5MGXFHVXH4QYBIO2XWLWOH5ANRM2NHYA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.555541803 (432h0m52.383250199s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.561754672 (432h0m52.389463063s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.565484488 (432h0m52.393192887s) [noise=true , address=89.149.201.147:4008 , piece_id=2FME7O25WWJGPW3QR7SNATQXK4UA2KZIN7J2ZUNRQTM67NVR2Y2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.580597563 (432h0m52.408305955s) [noise=true , address=81.171.14.70:4027 , piece_id=FRRLBX6EMKUAEFNELKYS3WPNG6CDYX7WCBLVPHAW3AZGQBMB3V3A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596579728 (432h0m52.424288127s) [noise=true , address=81.171.14.70:4043 , piece_id=MJM5262FY7HLPJHDZU7Z5FOZKGQG6XLIEGHB3W3PECXJOUATGNBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596646083 (432h0m52.424354476s) [seg_pos={0 0}, upload=449474]: Segment reached optimal threshold of 26 pieces.
Dec 23 15:28:51.597463826 (432h0m52.425172227s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.597488049 (432h0m52.425196452s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598037697 (432h0m52.425746098s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.598080201 (432h0m52.425788602s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598169558 (432h0m52.425877955s) [seg_pos={0 0}, upload=449474]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 23 15:28:51.598361733 (432h0m52.426070137s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 15:28:51.598386842 (432h0m52.426095234s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 15:29:47.549935238 (432h1m48.377643635s) [upload=449474]: Done uploading segments...
Dec 23 15:29:47.550073441 (432h1m48.377781841s) [upload=449474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.050967606s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [....................................X......................X] duration:
1m5.793695818s trace:
https://jaeger.dev.storj.io/trace/20ee6d11665285b1
FAIL (2025-12-23 15:29:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450238296 (432h0m51.27794669s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450284693 (432h0m51.277993094s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.450928506 (432h0m51.2786369s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451455886 (432h0m51.279164277s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.451726821 (432h0m51.279435218s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452155821 (432h0m51.279864216s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.452617301 (432h0m51.280325695s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453031906 (432h0m51.280740306s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453464313 (432h0m51.281172704s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.453833854 (432h0m51.281542232s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Uploading piece...
Dec 23 15:28:50.873749393 (432h0m51.701457785s) [upload=449474]: Next returned nil segment
Dec 23 15:28:50.873762459 (432h0m51.701470849s) [upload=449474]: Waiting for error group managing segments...
Dec 23 15:28:51.042505138 (432h0m51.870213541s) [noise=true , address=81.171.14.70:4036 , piece_id=T6353GQC3VCIXARGB7JB66ROB4HXB54ZJEN2NLZQZSCXGFAVPBBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.047157036 (432h0m51.874865439s) [noise=true , address=81.171.14.70:4039 , piece_id=5JFM3FGX4UXUKJDDUR6QKPEO2Q6PQXICVFIHLMCYUCHM3WRBD4GA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.061283412 (432h0m51.888991827s) [noise=true , address=185.167.204.206:4066 , piece_id=TXAR7UAXS4MKKGJHLSTE7KW53JKPV6BZVMUXMTPATMYM3SLKOGGQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.062400429 (432h0m51.89010882s) [noise=true , address=89.149.201.147:4005 , piece_id=CJBMKELGKRI6W72QLOL2PWTLYE47U224EEC34ELC5VVMIU6Y3GYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.066410014 (432h0m51.894118413s) [noise=true , address=185.167.204.205:4009 , piece_id=QOWW2X6YAMSGYW3BC7ZRPNLZ6ZZJ3CQVSYACJDWHLJNAMDMBQPBA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.095270986 (432h0m51.92297939s) [noise=true , address=185.167.204.206:4047 , piece_id=ONFTXH7MBLSGVMBOQUKM2SLMAEJ52TZIEDNABNC22D7VXVPCXY5Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.100596666 (432h0m51.928305061s) [noise=true , address=185.167.204.205:4019 , piece_id=JAQWQNVVW7DUV5H7O52ILYHN55VVWADGGAYFBDK4VM7AZWNAXCRQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.103560142 (432h0m51.93126853s) [noise=true , address=81.171.14.70:4025 , piece_id=WGTWZVCDY4N2FGSOCKQEDS5MC25AHPI2MFBF2VRO47ZF5E65CY2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.118031585 (432h0m51.945739979s) [noise=true , address=185.167.204.205:4021 , piece_id=7D7PQWUUQ6OV66VP73XGLAURNLQFOX5YT3MURRQSAN66VKA3DKYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.128062622 (432h0m51.955771018s) [noise=true , address=185.167.204.206:4051 , piece_id=LH4GXQ2MP4BXROJBXAFHEWJZMGSBRWNBITLF37NNFCDLB7IEPQLA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.134397721 (432h0m51.962106124s) [noise=true , address=89.149.201.147:4009 , piece_id=NLGLDZI6VRLXIP2S3G6E6BCK6ERGF3LKIHGKKYDRTDU7IAE3MOEA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.163106517 (432h0m51.990814919s) [noise=true , address=185.167.204.206:4054 , piece_id=G4Q6U5WSHNOBV5YE3HVZFZODWX3NJ2J67WATNLF47ZJVAIDR7CNQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.187215887 (432h0m52.014924285s) [noise=true , address=185.167.204.205:4005 , piece_id=SMS6QSW26MFG6ZWXS43SJNVKW5L2KC2HZVIB5ZN6ARTZE7VCDXFQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.191385548 (432h0m52.019093943s) [noise=true , address=185.167.204.205:4025 , piece_id=STHOVH3EE4E5EGHV3YXD7PAK7HL64A4YZX5I2L7REVBDM7SOXG4A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.232740501 (432h0m52.060448892s) [noise=true , address=185.167.204.206:4067 , piece_id=A5Z7XL6JY37OWKH3ZDC5A2W73ERAQCS2WIOXASUHEX3CPYCG2PYQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.254566742 (432h0m52.082275133s) [noise=true , address=185.167.204.206:4052 , piece_id=KWPFK2SOTESIA4YWLNPM6CES7DUPWCSVAI3E5UDYK6YSNM3EFEOA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.292399281 (432h0m52.120107676s) [noise=true , address=185.167.204.206:4071 , piece_id=SLOZ43XJOYRWOK4FHBYSQFFLIZMHVOHKMVIDJL37GXVT2JRDPWKQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.495257157 (432h0m52.322965538s) [noise=true , address=185.167.204.205:4018 , piece_id=XAIHJLWSUIGGHLKETNCWUAUCA76QOHNN3BD6H3H47XQMONNHDVCQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.509483881 (432h0m52.337192272s) [noise=true , address=81.171.14.70:4035 , piece_id=YIAIQHW5HLQS63JX6TITC3YZKBNH3BC5YYJ7DNC7EWCXL6ILDHOQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.519317972 (432h0m52.347026369s) [noise=true , address=81.171.14.70:4033 , piece_id=4RARLBHMGEOWN6SS2DKQ6DVWSUO54G4LV5UP2LYOGJAZDEEWD2CA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.540953811 (432h0m52.368662212s) [noise=true , address=89.149.201.147:4017 , piece_id=YKTOFL6L2A2OSVXSM2NW5MGXFHVXH4QYBIO2XWLWOH5ANRM2NHYA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.555541803 (432h0m52.383250199s) [noise=true , address=81.171.14.70:4040 , piece_id=IYET6GALTRUFQX34RVXLM4WUVWTKGP2HAATO3DRNBBOC7RX4BPDA, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.561754672 (432h0m52.389463063s) [noise=true , address=89.149.201.147:4020 , piece_id=PMCD7ZHETP3AEJBEZT2GUWMMQBQDW52T2AGQINLWB63O4IQ7PSWQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.565484488 (432h0m52.393192887s) [noise=true , address=89.149.201.147:4008 , piece_id=2FME7O25WWJGPW3QR7SNATQXK4UA2KZIN7J2ZUNRQTM67NVR2Y2Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.580597563 (432h0m52.408305955s) [noise=true , address=81.171.14.70:4027 , piece_id=FRRLBX6EMKUAEFNELKYS3WPNG6CDYX7WCBLVPHAW3AZGQBMB3V3A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596579728 (432h0m52.424288127s) [noise=true , address=81.171.14.70:4043 , piece_id=MJM5262FY7HLPJHDZU7Z5FOZKGQG6XLIEGHB3W3PECXJOUATGNBQ, seg_pos={0 0}, upload=449474]: Done uploading piece. err: <nil>
Dec 23 15:28:51.596646083 (432h0m52.424354476s) [seg_pos={0 0}, upload=449474]: Segment reached optimal threshold of 26 pieces.
Dec 23 15:28:51.597463826 (432h0m52.425172227s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.597488049 (432h0m52.425196452s) [noise=true , address=89.149.201.147:4024 , piece_id=KNY2LKNZAIKFSIHLQ4B7EY766SW4HYVRQY6ZOSOYPH2OZFSRB65Q, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598037697 (432h0m52.425746098s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T7zKyoeXVwkgTKXkQqvXi28sWXHE9vRdcNYg7xR8tWFxdtDqf): protocol: expected piece hash; context canceled; EOF
Dec 23 15:28:51.598080201 (432h0m52.425788602s) [noise=true , address=81.171.14.70:4032 , piece_id=JHQOOLIUO5ILW6JQQW5OTTKNOHIMH5XYDMQ4MHUAPYWFJX2UH57A, seg_pos={0 0}, upload=449474]: Long tail cancellation due to optimal threshold
Dec 23 15:28:51.598169558 (432h0m52.425877955s) [seg_pos={0 0}, upload=449474]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 23 15:28:51.598361733 (432h0m52.426070137s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 15:28:51.598386842 (432h0m52.426095234s) [upload=449474]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 15:29:47.549935238 (432h1m48.377643635s) [upload=449474]: Done uploading segments...
Dec 23 15:29:47.550073441 (432h1m48.377781841s) [upload=449474]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m5.793717393s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration:
52.086482416s trace:
https://jaeger.dev.storj.io/trace/a5ae6135be9170
FAIL (2025-12-23 15:29:33 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 52.086509466s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
269.892942ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:29:29 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 269.916973ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X..............X.....] duration:
8.226527841s trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:29:27 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 8.226560882s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [........................................X..............X....] duration:
869.224974ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:28:29 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 869.251011ms
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
258.773431ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:28:15 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 258.814395ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.528690073s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:28:11 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.528796174s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X..............X...] duration:
864.97837ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:28:07 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 865.001628ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.804810924s trace:
https://jaeger.dev.storj.io/trace/57972501076f255c
FAIL (2025-12-23 15:28:04 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.804840164s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
261.786562ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:27:48 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 261.809852ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.612418294s trace:
https://jaeger.dev.storj.io/trace/57972501076f255c
FAIL (2025-12-23 15:27:28 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.612449763s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.478309662s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:27:27 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.478337622s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X..............X..] duration:
897.690944ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:27:16 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 897.716448ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.050455502s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:27:04 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.050490314s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.726750551s trace:
https://jaeger.dev.storj.io/trace/57972501076f255c
FAIL (2025-12-23 15:26:44 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.726778949s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
262.48714ms target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:26:39 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 262.511347ms
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.082726236s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:26:33 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.082754685s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.673234912s trace:
https://jaeger.dev.storj.io/trace/57972501076f255c
FAIL (2025-12-23 15:26:27 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 2.67325464s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X..............X.] duration:
816.145766ms trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:26:19 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 816.171345ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.085416713s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:26:13 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.085441283s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
25.646269929s target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:26:11 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 25.646310103s
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
29.602092634s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:26:08 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 29.602124811s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
14.765990123s trace:
https://jaeger.dev.storj.io/trace/57972501076f255c
FAIL (2025-12-23 15:26:07 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 14.766020225s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001294257s trace:
https://jaeger.dev.storj.io/trace/57972501076f255c
FAIL (2025-12-23 15:25:52 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001320467s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
2 out of 60 is failed (threshold=3) [............................................X..............X] duration:
30.00106401s trace:
https://jaeger.dev.storj.io/trace/4115cae1c19ca6ae
FAIL (2025-12-23 15:25:48 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001096843s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001489822s target:
https://eu1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/3c8a15cd801ebdc1
FAIL (2025-12-23 15:25:45 UTC)
orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout
Duration: 30.001511056s
core
satellite
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.063608231s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:24:30 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.063675453s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.053300063s trace:
https://jaeger.dev.storj.io/trace/7d01e6505efb6bc5
FAIL (2025-12-23 15:24:14 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 2.053327884s
core
uplink
eu-select-1
eu1-placement-15
|