Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
1.667391193s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:56:01 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.66752529s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
1.308928456s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:55:29 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.308961558s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
48.571633ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:55:28 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 48.58894ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
1.280523956s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:55:16 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.280555248s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
33.538764ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:55:12 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 33.55871ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
1.42743093s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:55:00 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.427529446s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
1.51121195s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:54:34 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.511254136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
1.334342781s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:54:17 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.334439021s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
31.838655ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:54:02 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 31.871879ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
1.320789514s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:53:31 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.320815832s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
1.359645059s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:53:20 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.359673489s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
35.598462ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:52:56 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.641476ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
1.519931122s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:52:50 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.519985396s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
1.517042282s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:52:27 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.517085381s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
1.34019947s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:52:11 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.340315579s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
1.277746658s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:52:06 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.277775528s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
31.165179ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:51:49 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 31.234633ms
core
satellite
us1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
29.944416ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:51:23 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 29.976518ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
1.384987986s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:51:18 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.385025415s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
1.285751918s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:51:13 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.285785182s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
37.824592ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:50:41 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 37.852902ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
1.446635516s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:50:19 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.446671501s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
1.365585176s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:50:13 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.365690594s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
32.475378ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:49:32 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 32.506881ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
1.444580646s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:49:16 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.444654558s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
1.432901186s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:49:15 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.432994835s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
1.356635586s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:48:50 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.356760298s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
30.555952ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:48:22 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 30.591937ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
1.420701757s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:48:18 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.420759451s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
1.284047758s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:47:51 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.284082197s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
31.2994ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:47:33 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 31.33793ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
1.385836309s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:47:31 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.385898004s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
1.288651145s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:47:00 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.288684197s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
32.631484ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:46:59 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 32.661835ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
1.365575753s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:46:57 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.365678315s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
1.398071425s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:46:21 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.398199291s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
36.533346ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:46:13 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 36.578965ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
1.244376396s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:45:46 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.246565007s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
36.263386ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:45:33 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 36.32323ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
1.346398278s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:45:20 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.34643256s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
29.223475ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:45:13 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 29.248825ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
1.46639363s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:45:04 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.466714175s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
1.348859115s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:44:54 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.348934872s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
1.36152915s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:44:34 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.361758036s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
35.718825ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:44:23 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.816693ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
1.686992168s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:43:58 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.687039081s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
42.544821ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:43:53 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 42.570891ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
1.338895881s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:43:30 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.338926308s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
31.194873ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:43:23 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 31.230115ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
1.294008188s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:43:08 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.294041138s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|